Nvidia Dynamo Turns Kubernetes into an Inference Platform

A single model server makes everyone happy in a small demo: you open an endpoint, send requests, get answers back. Once the team grows and traffic gets serious, the picture changes; the same endpoint has to run on a cluster, multiply, spread requests across the right workers, and stay up when parts fail. Moving an AI service into production means putting these invisible chores in order more than growing the model. That is exactly where this walkthrough begins: a team outgrowing one server, heading for an OpenAI-compatible endpoint on Kubernetes.

On Kubernetes, Dynamo follows the operator pattern: the team writes down the setup it wants in a manifest file, the cluster stores that desired state, and the operator works to bend the live setup toward it. The mental picture has four moves: write the deployment request, apply it to the cluster, hand the pieces to the operator, then send traffic to the OpenAI-compatible endpoint. The appeal is that deployment stops being a pile of handmade scripts and becomes a repeatable cluster chore. When the serving design changes, you update the request and reapply; the operator carries the rest. For an AI service with a living model lineup, that loop is the whole game.

Dynamo offers the team two paths. The first starts with a DGDR, a statement of intent: the team names the model, the runtime engine, the workload, the accelerator pool, and any service targets, and Dynamo profiles the setup and produces a ready recipe called a DGD. The Next path starts directly with a DGD: you take a proven recipe and bend the inference graph and serving layout to your needs. In both paths the operator turns the DGD recipe into the running parts that serve traffic. The Nvidia docs quickstart tells this flow through apply, wait, and probe moves with kubectl.

A client sends its request to the Dynamo frontend, which serves an OpenAI-compatible endpoint, so the familiar request shape on the client side never changes. Behind that boundary Dynamo routes the request to backend workers, each serving the model with a runtime engine such as SGLang, vLLM, or TensorRT-LLM, as the recipe dictates. This split matters: the client keeps seeing one steady interface while the serving arrangement behind it keeps evolving. A team starting small maps one model, one engine, and one worker shape into the request; when more capacity is needed, it updates the deployment instead of rebuilding the whole stack.

Before Dynamo, the team lives with one model server: easy to start, yet as needs grow the team must solve deployment, discovery, scaling, routing, endpoint exposure, and observability on its own. After Dynamo, a cluster-native serving layer takes over: the deployment request states what is wanted, the operator owns the lifecycle, the frontend stays familiar, and workers shape around model and engine picks. The Nvidia docs walkthrough of automatic deployment shows the finer moves of the generated recipe, from profiling runs to service targets.

The live demo keeps the DGDR flow deliberately plain. First the request is written with model, engine, workload, accelerators, and service targets. Next the file goes onto the cluster with kubectl. Then the team waits while the operator builds the parts. Finally a request hits the endpoint and an answer comes back. The recipe-first road ends at the same door: take a ready DGD recipe, apply it with kubectl, and let the operator turn the inference graph into running parts. Either way the outcome is the same: a repeatable cluster chore instead of a handmade setup.

This walkthrough fits teams about to move a small endpoint onto a cluster: mapping model, engine, and worker settings into one deployment request is the cleanest start. The ai-dynamo repository on GitHub hosts the Apache 2.0-licensed software with its Helm charts; the operator and definition files for a first install come from there. The Nvidia product page places Dynamo as production-scale inference serving and points readers who want depth toward the docs.