Use this when you have (or are about to ship) AI-powered features and need to monitor them in production. Covers LLM-specific concerns that traditional observability and SRE monitoring miss: prompt/response quality, token costs, latency per model, hallucination drift, and model version regression. If you're looking to measure user behavior, use /observability-plan. If you're looking to measure system health, use /instrumentation-plan. This skill covers the AI-specific layer between those two.

The distinction: /observability-plan answers "Are users successful?" /instrumentation-plan answers "Is the system healthy?" This skill answers "Is the AI behaving correctly, consistently, and affordably?"

Related skills: Complements /observability-plan (product analytics) and /instrumentation-plan (SRE metrics). Eval criteria from /ai-eval-design become monitoring thresholds here. Quality dimensions from /ai-health-check CARATS framework inform what to monitor.

Process

Step 1: Identify LLM touchpoints

Ask the user:

1. What AI features are in production (or about to ship)? (List each feature that calls an LLM)
2. Which models does each feature use? (Provider, model name, version)
3. What's the current monitoring? (Any logging, dashboards, or alerts already in place?)
4. What problems have you seen? (Quality issues, cost surprises, latency spikes, outages)
5. What's the sensitivity level? (Can you log prompts/responses, or are there PII/compliance constraints?)
6. What observability tools are in use or available? (Helicone, Langfuse, Datadog, custom, etc.)

Map each LLM touchpoint:

Feature	Model	Calls/day	Avg latency	Current monitoring	Known issues
(Chat support)	(Claude Sonnet)	(10,000)	(2.1s)	(None)	(Occasional hallucinations)
(Doc summary)	(Gemini Pro)	(500)	(8.3s)	(Basic logging)	(Slow on large docs)

Step 2: Define quality metrics

Quality monitoring answers: "Is the AI output still good?" Define metrics that catch degradation:

Metric	Definition	How to measure	Threshold	Alert when
Consistency score	Same input produces similar outputs over time	Run golden dataset weekly, compare scores	> 3.5 rubric avg	Score drops below 3.0
Hallucination rate	% of outputs containing fabricated information	Automated fact-check or LLM-as-judge sampling	< 5%	Rate exceeds 10%
Relevance score	% of outputs that address the user's actual question	LLM-as-judge on sample + user feedback signals	> 85%	Below 75%
Tone compliance	Output matches expected voice/style	Tone rubric scoring on sample	> 90% pass	Below 80%
Safety incidents	Harmful, biased, or inappropriate outputs	Content filter + human review of flagged items	0 critical	Any critical incident
User satisfaction signal	Thumbs up/down, regeneration rate, copy rate, conversation abandonment rate	In-product feedback + behavioral tracking (did the user accept, edit, or discard the output?)	(baseline)	Drops > 20% from baseline

Not every feature needs every metric. Match metrics to the feature's risk profile:

• Customer-facing, high-stakes: All metrics, tight thresholds
• Internal tool, moderate stakes: Consistency + relevance + cost
• Batch processing, low stakes: Cost + basic error rate

Step 3: Design the logging strategy

What to capture on every LLM call:

Field	Type	Purpose	PII concern?
request_id	string	Unique identifier for the call	No
session_id	string	Conversation or session ID (for multi-turn features)	No
timestamp	ISO 8601	When the call was made	No
model	string	Model name and version	No
feature	string	Which product feature triggered this	No
input_tokens	integer	Token count of the prompt	No
output_tokens	integer	Token count of the response	No
latency_ms	integer	Total response time	No
time_to_first_token_ms	integer	Streaming start time	No
status	string	Success, error, timeout, rate_limited	No
error_type	string	Error category if failed	No
prompt_hash	string	Hash of the system prompt template (not content)	No
cost	float	Calculated cost of this call	No
tool_calls	JSON	Tool/function calls made during the request (if any)	Depends on tool
prompt_text	string	Full prompt (if allowed)	Yes -- may contain PII
response_text	string	Full response (if allowed)	Yes -- may contain PII
user_id	string	Anonymized user identifier	Yes -- handle carefully

PII handling decisions:

• Can you log full prompts and responses? (Best for debugging, worst for privacy)
• If not, can you log sanitized versions? (Strip PII, keep structure)
• If not, can you log metadata only? (Tokens, latency, cost -- no content)
• What's the data retention policy? (30 days? 90 days? Indefinite?)
• Who has access to raw logs vs. aggregated dashboards?

Sampling strategy for high-volume features:

• Log metadata (tokens, latency, cost, status) on 100% of calls
• Log full prompt/response on 1-10% of calls (configurable)
• Log full prompt/response on 100% of error/timeout calls
• Run quality eval (LLM-as-judge) on 1-5% sample

Step 4: Plan cost and latency monitoring

Cost monitoring:

Metric	Formula	Dashboard	Alert when
Cost per interaction	(input_tokens x input_price + output_tokens x output_price)	Real-time	> 2x baseline
Daily cost by feature	Sum of interaction costs per feature per day	Daily	> budget ceiling
Monthly cost projection	Daily cost x days remaining	Weekly	> monthly budget
Cost per user	Total LLM cost / active users	Monthly	Trending up > 20% MoM
Token efficiency	Output quality score / tokens used	Weekly	Efficiency drops > 15%

Latency monitoring:

Metric	Target	Alert when
p50 response time	(target, e.g., < 2s)	> 1.5x target
p95 response time	(target, e.g., < 5s)	> 2x target
p99 response time	(target, e.g., < 10s)	> 3x target
Time to first token (streaming)	(target, e.g., < 500ms)	> 1s
Timeout rate	< 1%	> 3%
Rate limit hit rate	< 0.1%	> 1%

Step 5: Design drift and regression detection

Model behavior changes over time -- from model updates, prompt changes, data shifts, or provider-side changes:

Drift detection approach:

Signal	What changes	How to detect	Frequency
Model version change	Provider updates the model	Monitor model version in API responses	Every call
Output distribution shift	Average output length, vocabulary, structure changes	Statistical comparison of output properties week over week	Weekly
Quality regression	Eval scores drop	Run golden dataset eval, compare to baseline	Weekly
Cost drift	Token usage changes without prompt changes	Compare avg tokens per call week over week	Daily
Latency drift	Response times change	Compare p50/p95 week over week	Daily
Prompt template change	Team modifies system prompts	Track prompt_hash, alert on changes, require eval rerun	On change

Regression response playbook:

1. Alert fires -- quality score dropped or cost spiked
2. Triage -- is this a model version change, prompt change, or data change?
3. Compare -- run golden dataset eval on current vs. previous model/prompt
4. Decide -- revert prompt, switch model version, adjust thresholds, or accept new baseline
5. Document -- log the incident and resolution for future reference

Step 6: Choose tooling

Tool	What it does	Best for	Pricing model
Helicone	LLM proxy with logging, cost tracking, caching	Teams wanting zero-code setup, cost optimization	Free tier + usage-based
Langfuse	Open-source LLM observability, tracing, eval	Teams wanting self-hosted or detailed tracing	Free (self-hosted) or cloud pricing
Braintrust	Eval platform with logging and experiments	Teams focused on systematic eval and prompt iteration	Usage-based
Datadog LLM Monitoring	Extension of Datadog APM for LLM calls	Teams already on Datadog	Per-host pricing
Arize Phoenix	Open-source LLM tracing and evaluation	Teams wanting self-hosted with strong eval integration	Free (self-hosted) or cloud pricing
LangSmith	LangChain's observability platform	Teams using LangChain/LangGraph	Free tier + usage-based
Custom (OpenTelemetry)	Roll your own with standard instrumentation	Teams with specific requirements or existing infra	Infrastructure cost

Selection criteria:

• What's your existing monitoring stack? (Extend it vs. add a new tool)
• Do you need self-hosted? (Compliance, data sovereignty)
• What's the budget for monitoring tooling?
• How many LLM calls per day? (Determines whether free tiers are viable)

Default recommendation for most teams: Start with Langfuse if you don't already have LLM monitoring in your stack. It covers tracing, eval, and cost tracking in one tool, offers both self-hosted and cloud options, and has the strongest open-source community momentum (5.0 rating, 41 reviews on Product Hunt as of March 2026). Move to Datadog LLM Monitoring only if your team already runs Datadog and wants unified APM + LLM observability.

Step 7: Generate the observability plan

Compile into a structured document:

# LLM Observability Plan: (Product name)

**Generated:** (date)
**Product:** (brief description)
**LLM features:** (count and list)
**Models in use:** (list with versions)

## LLM Touchpoint Map
(Table from Step 1 -- features, models, volume, current monitoring)

## Quality Metrics
(Table from Step 2 -- metrics, thresholds, measurement methods, alert rules)

## Logging Architecture
(Strategy from Step 3 -- what to log, PII handling, sampling rates)

### Log schema
(Field list with types and PII flags)

### Sampling rules
- Metadata: (100% of calls)
- Full content: (N% of calls, 100% of errors)
- Quality eval: (N% sample via LLM-as-judge)

### PII handling
- (Approach: full logging / sanitized / metadata only)
- (Retention policy)
- (Access controls)

## Cost & Latency Monitoring
(Tables from Step 4 -- cost metrics, latency targets, alert thresholds)

## Drift & Regression Detection
(Signals and playbook from Step 5)

## Tooling
(Recommendation from Step 6 with rationale)

## Implementation Checklist
- [ ] **(P0)** Instrument metadata logging on all LLM calls (tokens, latency, cost, status)
- [ ] **(P0)** Set up cost tracking dashboard with daily spend by feature
- [ ] **(P0)** Configure latency alerts (p95 > threshold)
- [ ] **(P1)** Implement prompt/response logging with PII handling
- [ ] **(P1)** Set up golden dataset eval as weekly automated run
- [ ] **(P1)** Build quality metrics dashboard
- [ ] **(P2)** Implement drift detection (output distribution monitoring)
- [ ] **(P2)** Create regression response playbook and runbook
- [ ] **(P2)** Set up model version change alerts

## Open Questions
- (Unresolved monitoring decisions)
- (Things that need baseline data to determine thresholds)

Step 8: Review and finalize

Ask the user:

• Are the quality metrics capturing what matters most for your AI features?
• Is the logging strategy practical given PII constraints?
• Are the cost alert thresholds realistic based on current spend?
• Is the drift detection approach proportionate to your risk tolerance?
• Does the tooling recommendation fit your existing stack and budget?
• Who owns LLM monitoring? (Engineering, ML team, product, SRE?)

Adjust based on feedback.

Output location

Present the plan as formatted text in the conversation for the user to copy into their product wiki, analytics documentation, or team shared drive.