← Back to Issue

In-house LLM serving at Netflix

From Programming Digest · subscribed via aiste.ulozaite@gmail.com · original ↗ · unsubscribe

Netflix runs its own LLM serving stack on top of vLLM and NVIDIA Triton, integrated into existing production infrastructure rather than a separate system. Key decisions around engine choice, model packaging, API design, and deployment strategies each revealed unexpected trade-offs only under real production load. A notable example is constrained decoding, where per-request CPU processing caused latency to grow linearly with batch size until a rewrite using vLLM V1’s batch-level API and C++ solved the bottleneck.

Couldn’t fetch the full article — read it on the original site ↗.