frontend: add /tokenize and /detokenize endpoints

[Pernekhan]

Adds POST /tokenize and POST /detokenize to the frontend, so a caller can ask what a prompt costs without running inference.

/tokenize takes either {"prompt": …} or {"messages": […]}; the latter renders the model's chat template first. /detokenize is the inverse. Both are system routes, appear in /openapi.json, and their paths are overridable via DYN_HTTP_SVC_TOKENIZE_PATH / DYN_HTTP_SVC_DETOKENIZE_PATH.

The number has to be the served number

The endpoints run the model's own OpenAIPreprocessor — the instance its generate pipeline was built with — rather than re-deriving tokenization. The watcher already constructed one per pipeline and moved it into build_pipeline; WorkerSet now retains it, Model and ModelManager expose it, and OpenAIPreprocessor grows the read-only entry points that stop before the worker hop:

• tokenize_chat — normalize, template, encode
• tokenize_completion — encode a bare prompt, which is all /v1/completions does
• detokenize / token_strings

normalize_chat_request is generate()'s own pre-template prologue hoisted into one function both callers share (default thinking mode, thinking aliases, named-tool-choice fixup). All of them mutate chat_template_args or tool_choice, so templating without them renders a different prompt.

An earlier revision re-derived that sequence instead and was wrong four separate times — render vs render_prompt, plain encode vs encode_segments, the skipped prologue, and NUL-byte stripping. Each was found by reading the code, not by a failing test, which is why the shared-instance approach is the one here.

Behaviour worth knowing

add_special_tokens defaults to false. /v1/completions applies no template and tokenizes without special tokens, so the prompt reaches the engine with no BOS; reporting anything else would report a count this deployment never produces. An explicit true returns 501.

These return 501 rather than an approximate answer: multimodal message parts, continue_final_message, a per-request chat_template, media_io_kwargs / mm_processor_kwargs, add_special_tokens: true, and a model with no Rust pipeline. Multimodal matters most — a chat template emits one placeholder where inference expands hundreds of tokens.

token_strs returns decoded text (" world") rather than vocabulary spellings ("Ġworld"). The latter is only reachable from an Encoding::Hf, and the prefix cache normalizes every encode to Encoding::Sp, so it would vary with DYN_TOKENIZER_CACHE.

Verification

Measured against the endpoints themselves — usage.prompt_tokens from the same frontend, model google/gemma-4-E4B-it:

/tokenize vs live /v1/completions	4 / 4 exact
/tokenize vs live /v1/chat/completions	4 / 4 exact
10 production tokenizers vs HuggingFace transformers	226 / 226
6 production models vs our deep-tokenizer service	30 / 30

Parity models: gemma-4-E4B-it, DeepSeek V3.1 / V3.2 / V4-Pro, GLM-4.6 / GLM-5, Qwen3-Coder-480B, Qwen3-235B-Thinking, Llama-3.1-8B-Instruct, Mistral-Nemo — both request forms, add_generation_prompt on and off, token_strs, and detokenize round-trips including CJK, emoji and accented text.

On this branch: cargo test -p dynamo-llm --no-fail-fast → 38 suites, 0 failures; clippy -D warnings, fmt --check and rustdoc all clean.

Three guards, each verified by injecting the drift it exists to catch:

• tokenize_chat_matches_the_generate_path_token_ids
• tokenize_completion_matches_the_completions_pipeline
• tokenize_completion_strips_null_bytes_like_the_generate_path

Also exercised end to end against a live python -m dynamo.frontend with etcd/NATS discovery and a mocker worker, on an image built the way dynamo-frontend:dyn141-v1 is built.

[Pernekhan]

too much commenting here