Production LLM Systems: What CTOs Need to Know

In short: a production LLM system is not a prompt wrapped in an API. It is a product system with retrieval, permissions, evaluation, monitoring, cost controls, fallback behaviour, and a workflow real users understand. Getting a good answer once is easy. Getting acceptable answers repeatedly, across messy inputs, changing data, latency limits and business constraints, is the actual job, and most of it is made of leadership decisions rather than model choices.

Why do LLM demos work and production systems struggle?

A demo proves the model can answer a question. Production asks different questions: how fast, how cheap, how often wrong, and how you find out when it is wrong. The gaps that appear between the two are predictable:

• Latency: a few seconds per answer is fine in a demo and unacceptable inside a workflow people run all day.
• Cost: a per-request price that looks trivial at a hundred users becomes a budget line at real volume.
• Accuracy: a small error rate is a curiosity in a demo and a stream of wrong decisions at scale.
• Context: your real source material does not fit in one prompt.
• Observability: without traces, nobody can explain why a specific answer was wrong.
• Change control: without versioning, every prompt edit is a live experiment on customers.
• Resilience: without fallbacks, a provider outage takes the product down with it.

None of these are solved by a better model. They are solved by engineering and by decisions a CTO has to make explicitly.

What should a CTO ask before a team builds one?

Four grounding questions do most of the work:

• What is the source of truth? Which documents, records or systems does the model answer from, and who owns keeping them current?
• How do we know an answer is good? What labelled examples exist, and what score means "ready"?
• What happens when confidence is low? Which cases go to a person, and who is that person?
• How will engineers debug a failure after launch? What is logged, and can a single bad decision be replayed?

Teams that answer these explicitly build systems that improve. Teams that skip them end up with impressive demos that are hard to support, measure or sell.

RAG or fine-tuning?

For domain knowledge, retrieval-augmented generation (RAG) is the sensible default. Your policies, contracts and internal records are not in any model's training data, and they change. RAG keeps knowledge in a store you control, lets you update it without retraining, and lets you show which source an answer came from. Fine-tuning earns its place when you need a consistent style, format or narrow behaviour that prompting and retrieval cannot deliver. Treat it as a later optimisation, not the starting point.

Two retrieval details matter more than teams expect. First, rerank: vector similarity is not relevance, and a prompt stuffed with loosely related chunks distracts the model. Retrieve broadly, then keep only the few passages that actually answer the question. Second, ground the answer: require citations to retrieved sources, and treat an answer that cannot cite as an answer that needs review.

One model or several agents?

Complex decisions often go better split into steps: a router that classifies the request, specialists that handle each kind, and a validator that checks the result before it leaves the system. Splitting the work makes each step easier to evaluate and cheaper to run, because routine steps can use smaller, faster models. The cost is more moving parts to observe. Choose it when a single prompt has become a tangle of special cases, not before.

How do you keep it accurate?

Assume the system will sometimes be confidently wrong, and design for it:

• Confidence thresholds: have the system score its own certainty and route low-confidence cases to human review instead of acting on them.
• Ground-truth sets: keep a labelled set of real cases, including the awkward ones, and score every change against it.
• Red-teaming: deliberately try to make the system invent citations or misapply a rule before customers find those cases for you.
• Prompts are code: version them, test them against the ground-truth set before release, roll them out gradually, and keep an instant rollback.

What does "observable" mean for an LLM system?

Log every model call with its inputs, retrieved context, output, latency and cost, tied to the request that triggered it. Watch latency percentiles, error rates, cost per hour and traffic in real time. Review a sample of decisions regularly, along with the human-review rate and user corrections. When something breaks, the question "why did it decide that?" should have a concrete answer.

A pre-launch checklist

• Accuracy measured on a labelled set that includes edge cases, against a target the business agreed to
• Load-tested well beyond expected traffic, with p95 and p99 latency recorded
• Cost per request measured and projected at realistic volumes, and judged against the value of the decision
• Adversarial testing for invented citations and wrong reasoning
• Gradual rollout and instant rollback for prompt and model changes
• Confidence thresholds and a staffed human-review path
• Tracing and structured logs live from day one
• Architecture decisions written down: why RAG, why this model, why this store

Does this pay off?

It can, when it is measured. At Uber I built ELLE, an AI decision engine for privacy reviews that cut manual review time by roughly 70% across millions of decisions. In systems like that, the model matters less than the discipline around it: clear sources of truth, measurable quality, a human path for uncertain cases, and the ability to explain a decision after the fact.

That is the leadership lesson. Production AI is a set of decisions about trust, accountability and cost that happen to be implemented in software. They belong to the CTO. This essay draws on the production AI chapter of The Centaur CTO.

Back to writing