Topic 8: LLMOps interview questions

LLMOps is the engineering discipline for operating applications built with large language models. It covers prompt and model versioning, data and evaluation pipelines, deployment, observability, cost control, security, and incident response. Unlike ordinary software operations, outputs are probabilistic and can change when a provider updates a model. A strong LLMOps process makes behavior measurable, releases reversible, and quality, latency, safety, and spend visible together.

I would keep prompts, model identifiers, parameters, tools, retrieval settings, and output schemas in version-controlled configuration. Each production request should record a non-sensitive configuration version and model response metadata. Changes should pass automated evaluation and, for important flows, a canary or shadow rollout. Feature flags make rollback fast. Prompt text should be reviewed like code because a small wording change can alter safety, cost, and business behavior.

It should contain representative test cases, expected properties, automated graders, human review for ambiguous samples, and regression reporting. Tests should cover normal requests, edge cases, adversarial prompts, refusals, tool calls, retrieval grounding, and structured output. I would segment results by language, customer type, task, and difficulty. The pipeline should compare candidate and baseline versions, fail on agreed thresholds, and preserve failed examples for future regression tests.

Hallucination needs a task-specific definition. For grounded question answering, I would measure whether claims are entailed by retrieved or supplied evidence, using an automated judge plus sampled human verification. For extraction, compare fields with labeled truth. I would track unsupported-claim rate, citation correctness, abstention quality, and severity, not only one aggregate score. Evaluation data must include questions whose answer is absent so confident guessing is detected.

I would monitor request volume, success and timeout rates, latency by stage, token usage, cost, model and prompt versions, tool errors, retrieval quality, schema failures, and safety outcomes. Traces should connect a user request to retrieval, model calls, tool calls, and the final response. Raw prompts may contain sensitive data, so logging needs redaction, access controls, sampling, and configurable retention rather than indiscriminate capture.

First measure cost by workflow, tenant, model, and token type. Then reduce unnecessary context, cap output length, summarize long histories, cache stable results, batch offline jobs, and route simple tasks to smaller models. Budgets and rate limits prevent runaway usage. I would monitor quality after every optimization because a cheaper answer that causes retries or human correction may have a higher total cost. Cost alerts should be actionable and tied to owners.

Prompt caching reuses processing for a repeated prefix or repeated request context, depending on the provider or infrastructure. It is useful for large stable system instructions, policies, or reference material that appears across many calls. Benefits include lower latency and input cost, but cache keys must reflect all content that affects behavior. I would define invalidation rules, avoid caching user-private data across tenants, and verify that cached content cannot become stale or unauthorized.

I would first run offline regression and safety evaluations, then deploy behind a feature flag. Shadow traffic can compare outputs without exposing them to users, followed by a small canary cohort with quality, latency, cost, and error monitoring. The release needs a known-good configuration, rollback procedure, and owner on call. Provider model changes should be treated as releases too, even when application code has not changed.

The application should use bounded timeouts, retries with exponential backoff and jitter, and idempotency for operations that may be repeated. A fallback model or deterministic path can handle supported low-risk tasks. If no safe fallback exists, return a clear temporary-unavailability message and preserve the user's work. Circuit breakers prevent a failing provider from consuming all capacity. Alerts should distinguish provider failures from application, network, and quota errors.

An LLM gateway is a shared service that manages access to one or more model providers. It can centralize authentication, routing, retries, rate limits, quotas, logging, redaction, caching, and policy enforcement. This reduces duplicated integration code and makes provider changes easier. The gateway must avoid becoming an opaque bottleneck: teams need clear ownership, tenant isolation, trace propagation, timeout budgets, and a documented path for provider-specific capabilities.

I would enforce limits at the tenant and workflow levels using token-aware or request-aware budgets. Queues smooth bursts, while concurrency limits protect downstream providers. Responses to quota errors should be classified separately from transient failures, because retries can worsen the problem. Usage dashboards and alerts should show remaining quota, projected exhaustion, and the responsible tenant. For critical workflows, reserve capacity or define priority rules with explicit fairness and degradation behavior.

Tools should have narrow schemas, least-privilege credentials, server-side authorization, and strict validation independent of the model. The model may propose an action, but application code decides whether it is allowed and whether confirmation is required. I would separate read and write tools, constrain destinations, sanitize outputs, and log arguments without leaking secrets. High-impact actions need idempotency, replay protection, and an auditable link between user intent and execution.

Retrieved documents are data, not instructions. I would separate system policy from retrieved content, label untrusted sources, limit tool permissions, and require the application to validate actions. Documents should be scanned for malicious patterns, but filtering is only one layer. The model should cite evidence and refuse instructions that conflict with policy. Red-team tests should include poisoned documents, indirect instructions, cross-tenant content, and attempts to exfiltrate system prompts or secrets.

Offline evaluation runs a fixed dataset before release, making candidate comparisons repeatable and inexpensive. Online evaluation measures behavior on real traffic through monitoring, user feedback, sampled review, or controlled experiments. Offline tests catch known regressions, while online signals reveal distribution shifts, new user intents, and operational issues. I would use both, with privacy-safe sampling and guardrails so an experiment cannot expose users to unacceptable safety or business risk.

Human review should target uncertainty and impact rather than every response. I would define clear labeling guidelines, show the source evidence and model version, and use double review for high-severity disagreements. Reviewers need escalation paths and periodic calibration. Sampling should include both suspected failures and random production outputs to measure detector blind spots. Their labels should feed evaluation sets, prompt improvements, routing thresholds, and incident analysis—not just a dashboard.

I would use provider-supported structured output or tool schemas where available, then validate every response with a strict schema. A bounded repair attempt may ask the model to correct only formatting, but it must not bypass business validation. If repair fails, route to a fallback or return a recoverable error. Metrics should distinguish invalid syntax, missing fields, invalid values, and downstream rejection so the actual failure mode can be fixed.

The process starts with detection, severity classification, and containment. I would freeze risky releases, disable a feature flag, reduce traffic, or switch models while preserving evidence. Then identify whether the cause is a prompt, model, data, retrieval, tool, provider, or infrastructure change. After recovery, run targeted regression tests, communicate impact, document the timeline, and add a durable prevention such as a test case, monitor, permission check, or rollback automation.

I would first establish whether the problem is missing knowledge, poor instructions, formatting, or a repeated behavior pattern. Retrieval is appropriate for changing factual content, while prompting and schemas often solve instruction and output issues. Fine-tuning may help with stable style, classification, or tool-selection behavior when sufficient high-quality examples exist. I would compare quality, data preparation, latency, cost, governance, and rollback complexity using the same evaluation set.

Governance should define what data may enter prompts, how it is classified, where it is stored, who can access it, and how long it is retained. Training and evaluation datasets need provenance, consent or licensing checks, redaction, deduplication, and access controls. Production logs should have separate retention policies from application records. I would document provider data-use terms, deletion procedures, audit ownership, and restrictions for regulated or confidential information.

Production readiness means the system is useful, measurable, safe, and recoverable. It needs representative evaluations, versioned prompts and models, schema validation, timeouts, bounded retries, rate limits, observability, privacy controls, cost budgets, and an incident runbook. There should be a tested rollback or fallback path and clear ownership. I would also verify user-facing failure messages, data deletion behavior, accessibility, and performance under realistic concurrency before launch.