LLMOps and AI Observability Engineer

Foundations of LLMOps and AI observability: differentiate MLOps, LLMOps and app observability, map the AI lifecycle (prompt, retrieval, tools, model, output, feedback), and set SLOs for quality, latency, cost and safety using logs, traces and metrics.

Clarifies how LLMOps differs from traditional MLOps and general app observability.

Details the AI lifecycle: prompt engineering, retrieval, tool integration, model inference, output handling and feedback loops.

Introduces SLO design that treats quality, latency, cost and safety as measurable dimensions.

Explains how logs, traces and metrics complement each other to monitor and debug AI systems.

Tracing AI workflows: instrument model calls (provider, model, tokens, latency), trace RAG retrieval and reranking steps, capture tool calls and agent loops, and apply redaction to protect privacy in traces.

Instrument and log model calls with provider, model version, token usage and latency metadata.

Trace RAG pipeline steps—retrieval and reranking—to establish provenance and relevance diagnostics.

Capture tool invocations and agent loops to detect unwanted behavior or runaway interactions.

Redaction techniques and best practices to protect privacy and limit leakage of sensitive data in traces.

Quality monitoring: online and offline evaluation strategies, prompt and version tracking, feedback collection and annotation workflows, and regression gates before deployment to prevent performance degradation.

Compares online versus offline evaluation approaches to measure AI output quality in context.

Covers prompt and component version tracking to correlate changes with behavior shifts.

Outlines feedback collection and annotation pipelines to generate labeled signals for monitoring.

Explains regression gates used to ensure no quality regressions are introduced prior to deployment.

Operations and incident handling for LLM systems: this module addresses monitoring token cost spikes and latency, fallback routing for provider outages, safety-incident triage, and postmortems aimed at continuous reliability improvements for AI services in production.

Detecting and responding to token cost spikes and latency incidents; techniques to identify root causes and mitigate impact.

Designing provider fallback and routing strategies with clear criteria for failover and traffic redistribution.

Triage workflow for safety incidents (e.g., harmful outputs, hallucinations) and guidance on incident prioritization.

Postmortem practice and continuous improvement loops to capture lessons and strengthen AI observability.