LLMOps
Last reviewed: August 2026 | This is a fast-moving area subject to quarterly review.
Overview
Section titled “Overview”After selecting a model (AI Platforms) and building a RAG pipeline (Advanced RAG Patterns), you need to continuously maintain and improve quality in production. This is LLMOps.
graph LR
A[Prompt Authoring] --> B[Evaluation] --> C[Deployment] --> D[Monitoring] --> E[Improvement]
E --> A
Evaluation
Section titled “Evaluation”Offline Evaluation
Section titled “Offline Evaluation”Verify quality before deployment.
| Type | Method | Tools |
|---|---|---|
| Golden Set | Measure accuracy against test set with known answers | Custom + auto-scoring |
| LLM-as-Judge | Another LLM evaluates response quality | Bedrock Evaluations, Vertex AI Eval |
| Human Review | Humans review samples | Labeling tools (Label Studio, etc.) |
| Regression Test | Confirm existing quality after prompt/model change | CI pipeline integration |
RAG Evaluation Metrics
Section titled “RAG Evaluation Metrics”RAG retrieval/response quality metrics (Recall@K, MRR, NDCG, Faithfulness, Answer Relevance, Context Precision/Recall) and evaluation tools are covered in detail in Advanced RAG Patterns — Evaluation.
Vendor Evaluation Tools
Section titled “Vendor Evaluation Tools”| Vendor | Service | Characteristics |
|---|---|---|
| AWS | Bedrock Evaluations | Auto + human eval, model comparison |
| Azure | Azure AI Evaluation SDK | Python SDK, CI/CD integration |
| Google Cloud | Vertex AI Evaluation Service | Auto metrics + human eval |
| Vendor-neutral | Ragas, DeepEval | Open-source RAG evaluation frameworks |
Prompt/Model Version Management
Section titled “Prompt/Model Version Management”| Subject | Method | Tools |
|---|---|---|
| Prompt versions | Manage prompt templates in Git. Auto-run eval on change | Git + CI |
| Model versions | Pin model ID/version in code. A/B test on upgrade | Bedrock Model ID, Azure Deployment |
| Deployment strategy | Canary (10% traffic to new version) → full rollout | Routing config |
| Rollback | Instant revert to previous prompt/model version | Deployment pipeline |
Operational Metrics (Monitoring)
Section titled “Operational Metrics (Monitoring)”| Metric | Meaning | Alert Threshold Example |
|---|---|---|
| Latency (p50/p99) | Response time | p99 > 5s |
| Token Usage | Input/output token consumption | >200% of daily average |
| Error Rate | API error ratio | > 1% |
| Cache Hit Rate | Prompt caching effectiveness | < 50% (below expectations) |
| Cost per Request | Per-request cost | Budget exceeded |
| Fallback Rate | Primary model failure → fallback model | > 5% |
Vendor-specific monitoring:
- AWS: CloudWatch + Bedrock metrics (InvocationLatency, InputTokenCount, OutputTokenCount)
- Azure: Azure Monitor + AI Studio metrics
- Google Cloud: Cloud Monitoring + Vertex AI metrics
LLM Observability Platforms
Section titled “LLM Observability Platforms”Beyond vendor-native monitoring, specialized observability tools exist for LLM workloads. They provide integrated prompt tracing, RAG quality analysis, cost tracking, and evaluation automation.
| Product | Type | Key Features | Notes |
|---|---|---|---|
| Arize AI | Commercial | Tracing, eval, drift detection, RAG analysis, guardrail monitoring | Phoenix (open-source version) |
| LangSmith | Commercial (LangChain) | LangChain/LangGraph native tracing & eval, prompt hub | Natural choice when already using the LangChain ecosystem |
| Langfuse | Open-source | Self-hostable, prompt management, tracing, cost tracking | Self-operable without vendor lock-in |
| Weights & Biases (Weave) | Commercial | Experiment tracking + LLM tracing + eval | Integrates with ML experiment management |
Selection criteria:
- Vendor-native tools (CloudWatch/Azure Monitor) cover basic metrics, but prompt-level tracing and RAG pipeline debugging require a dedicated tool
- LangSmith for LangChain-based stacks; Arize/Langfuse for framework-agnostic stacks
- Langfuse (self-hosted) or Arize Phoenix (open-source) when data sovereignty matters
Agent Observability
Section titled “Agent Observability”AI agents have multi-step trajectories (plan → tool call → observe → repeat), requiring different metrics than single LLM calls.
| Metric | Description | Why It Matters |
|---|---|---|
| Tool call success rate | Correct tool with correct parameters | Fastest drift signal |
| Trajectory length | Steps to task completion, retry count | Loop/inefficiency detection |
| Per-step latency | P50/P99 per stage (model inference, tool execution) | Bottleneck identification |
| Session cost | Tokens (per model) + tool call costs | Budget overrun early warning |
| Task completion rate | Goal achieved (success/failure/timeout) | Core business metric |
| Drift | Embedding/cluster shifts, behavior change after model version swap | Early detection of quality degradation |
| Online evaluation score | Production traffic sampling + LLM-as-Judge | Continuous quality assurance |
Agent observability tools:
| Tool | Agent-Specific Features |
|---|---|
| LangSmith | LangGraph trajectory replay, tool selection analysis, online evaluation |
| Datadog LLM Observability | Agent decision graph, loop detection, APM correlation |
| Braintrust | Trace → eval dataset → CI gate automation, Topics clustering |
| Arize AX | Continuous evaluation, trajectory accuracy, drift detection |
| Galileo | Luna evaluators (low cost/latency), tool selection quality, failure clustering |
| AgentCore Observability (AWS) | CloudWatch + OTEL auto-instrumentation, session/tool/memory metrics |
| Azure Foundry Monitoring (Azure) | OTEL-based multi-agent tracing, continuous evaluation, Azure Monitor integration |
| Vertex AI Tracing (Google) | Cloud Trace integration, ADK tracing, model + tool timeline |
| Langfuse (Open-source) | Self-hosted, framework-agnostic tracing, cost tracking, prompt management |
| Phoenix (Open-source) | OpenInference-based, self-hosted, trace + eval + drift |
Operational Patterns
Section titled “Operational Patterns”| Pattern | Description |
|---|---|
| Model Fallback | Auto-switch to backup model on primary failure/latency (e.g., primary Claude family → GPT family → Gemini family) |
| Rate Limit Handling | Queue or route to alternate provider when vendor rate limit reached |
| Budget Guardrail | Daily/monthly cost cap. Reject requests or switch to cheaper model on exceed |
| PII Masking | Auto-mask PII in prompt/response logs before storage |
Common Mistakes
Section titled “Common Mistakes”- Deploying prompt/model changes without evaluation — No regression testing → sudden quality degradation in previously-working responses.
- Storing PII in prompt/response logs unmasked — Privacy regulation violation and data breach risk.
- Single model dependency without fallback — Vendor rate limits or outages bring down entire service.
Checklist
Section titled “Checklist”- Golden Set regression tests run automatically in CI on prompt/model changes
- PII masking applied to all prompt/response logs
- Fallback strategy configured for automatic failover to alternate model
Related Documents
Section titled “Related Documents”- LLM Licensing and Cost Management — procurement and billing models (Seat/API tiers, reserved capacity, showback/chargeback)
- LLM Channel Selection Guide — differences between consuming the same FM 1P (direct) vs 3P (via cloud)
- AI Platforms and Model Comparison — model catalog, inference unit-cost optimization (caching, batch, routing)
- FinOps — cloud cost governance overall