Large language models broke the clean "train it, test it, ship it" model of production ML. The thing being operated is now a system that chains prompts, queries vector databases, and produces output judged on tone and safety — not just accuracy. That's LLMOps. And it's currently landing on top of your existing DevOps and MLOps workflows without a clear owner.
CNCF's Daniel Bryant has a clear take on who should own it — and the argument is sharper than it sounds.
"LLMOps doesn't need its own kingdom. It needs a well-run platform willing to let it in."
What actually changed
LLMOps isn't just MLOps with a new label. The gap is real:
- Scale and cost — LLMs cost substantially more to fine-tune and serve than classical models
- Fuzzier evaluation — accuracy scores don't capture safety, tone, or trustworthiness
- Ongoing ops — models drift, prompts stop working, integrations need constant tending
- New primitives — prompt versioning, vector stores, RAG pipelines, inference endpoints
MLOps teams have already built a parallel stack (MLflow, Kubeflow, Weights & Biases) because DevOps tooling never anticipated data versioning or drift monitoring. Without intervention, LLMOps becomes a third parallel stack, invisible to whoever governs the rest.
The lesson: shadow AI is the real risk
The bigger operational risk isn't a hallucinating chatbot. It's a team standing up its own RAG pipeline against an unreviewed vector store, outside any platform governance. Same pattern that made the DevOps-versus-platform split painful: a capability gets built outside the platform because the platform wasn't ready, and it never gets folded back in.
Bryant's framing via the CNCF Platforms Whitepaper is clean: model fine-tuning jobs, vector databases, prompt registries, and inference endpoints are just another platform capability. They need the same API, versioning, and ownership as anything else.
The tooling already exists in the CNCF ecosystem — Backstage at the product layer, Crossplane at the infrastructure layer, Kratix/KubeVela/KusionStack in the middle, exposing LLM pipelines through the same self-service interface as everything else.
What to do
If you're a platform engineer:
- Treat LLM infrastructure as a platform capability, not a data science side project
- Build a governed self-service path for inference endpoints, prompt deployments, and fine-tuning jobs before teams build their own
- Policy at request time: cost limits, data residency, model access controls — not discovered on the cloud bill
If you're on an MLOps or AI team:
- Push for your RAG pipeline and vector store to be first-class platform resources, not ad hoc infra
- Audit trail matters now — regulators want to know what changed, who approved it
If you're in platform leadership:
- The right question isn't "who owns the pipeline?" It's "who owns which layer, and is anyone coordinating across them?"
- Make the platform say yes fast, with governance built in — that's how you prevent shadow LLMOps
The CNCF TAG App Delivery Platforms Working Group is actively working on this. If your org is sorting out LLMOps ownership, it's worth following.
Source: CNCF Blog — LLMOps and platform engineering: Who should own the AI pipeline?
✏️ Drafted with KewBot (AI), edited and approved by Drew.
Top comments (0)