Focused on LLM inference / infra — currently digging into the internals of vLLM and SGLang, and contributing that work back to the community, mainly via vLLM-Omni and LMCache. Background in game development before that, with a C++ foundation and graphics experience.
Enjoys taking hardcore technical topics apart: wrote Understanding SGLang's Radix Cache, the LeetCode Way, building up to SGLang's Radix Cache implementation from classic LeetCode problems. Also translated Ray Tracing in One Weekend (C# implementation).
vLLM-Omni — working on correctness and performance of the MOSS-TTS inference pipeline.
| PR / RFC | What it does | Impact |
|---|---|---|
| #4157 | CUDA Graph capture/replay for the MOSS-TTS codec decoder, removing Python dispatch overhead | -7.6% mean E2E latency |
| #4230 | Batched matmul/embedding gather in the MOSS-TTS talker, replacing 2*n_vq serial kernel launches with 2 batched ops per decode step |
-39% latency (A6000); externally validated at -42% latency, 2.6x throughput (H20, concurrency 8) |
| #4415 | Fixed a cross-request audio corruption bug under batching — per-request list transport + corrected token-span row mapping for mixed prefill+decode | Correctness fix |
| RFC #4316 | Root-caused the batching bug above and designed the fix, plus a batched-decode dispatch and CUDA Graph bucketing proposal spanning four PRs | Design doc |
| #4220, #4264 | Repaired broken MOSS-TTS E2E tests and added benchmark coverage | Test/benchmark infra |
LMCache — contributed docs: #3432 added examples/README.md, indexing 25 example directories into four tiers (single-node basics → multi-instance sharing → production ops → advanced/ecosystem) to help new users find their way in.
Before moving into LLM inference, I worked in game development, doing C++ client and engine work at companies including Perfect World and iQIYI.