Instrumentation Plan

Use this when you need to design, audit, or improve system-level instrumentation for a product or service. This covers the SRE side of measurement: uptime, availability, latency, deployment health, error budgets, and alerting. If you're looking to measure user behavior, event tracking, or product analytics, use /observability-plan instead.

The distinction: Instrumentation answers "Is the system healthy?" Observability answers "Are users successful?" Both are needed. Start here if the system isn't reliably measurable yet.

For AI-powered features: This skill covers system-level instrumentation. If you need to monitor LLM-specific concerns (prompt quality, token costs, hallucination drift, model regression), use /llm-observability-plan.

Process

Step 1: Gather context

Ask the user to provide:

1. System description — what does this system do? (services, APIs, background jobs, static sites, workers, etc.)
2. Current instrumentation — what's already measured? (existing dashboards, alerts, logging, APM tools)
3. Infrastructure stack — hosting, CI/CD, CDN, databases, message queues, third-party dependencies
4. Deployment process — how code gets to production (manual, automated, frequency, rollback capability)
5. Incident history — recent outages, degradations, or close calls (what broke, how was it detected, how long to recover?)
6. Team context — who is on-call? Is there an existing SRE function or is this engineering-owned?

If the user doesn't have all of this, work with what's available. Flag gaps as assumptions.

Step 2: Define SLIs, SLOs, and error budgets

For each critical user journey or system capability, define:

Service Level Indicators (SLIs) — the measurable signals:

Service Level Objectives (SLOs) — the targets:

• Set SLOs based on user expectations, not aspirational perfection
• A 99.9% availability SLO means ~8.7 hours of allowed downtime per year
• Start conservative (lower targets), tighten as you build confidence
• Every SLO needs an explicit measurement window (rolling 30 days is standard)

Error budgets — the tolerance:

• Error budget = 1 - SLO (e.g., 99.9% SLO = 0.1% error budget)
• When budget is consumed, freeze feature releases and focus on reliability
• Track burn rate: how fast is the error budget being spent?

Present these in a table:

Step 3: DORA metrics baseline

Assess the team's delivery health using the four DORA metrics:

For each metric:

1. Current state — what's the team's actual performance? (Measure or estimate.)
2. Target state — where should they be in 90 days?
3. How to measure — specific data source (CI/CD logs, incident tracker, deployment pipeline)
4. Biggest blocker — what's preventing improvement?

Step 4: Infrastructure health metrics

Define the standard infrastructure signals to monitor:

The Four Golden Signals (per Google SRE):

• Latency — time to serve a request (distinguish successful vs. failed request latency)
• Traffic — demand on the system (requests/sec, sessions, transactions)
• Errors — rate of failed requests (explicit 5xx, implicit timeout, wrong-answer errors)
• Saturation — how full the system is (CPU, memory, disk, queue depth, connection pool)

For each service or component, produce a monitoring table:

Step 5: Alerting strategy

Design alerts that are actionable, not noisy:

Alert tiers:

Alert design rules:

• Every alert must have a documented response action (or it's noise)
• Use burn-rate alerts for SLO monitoring (not raw threshold alerts)
• Set alerts on symptoms (user impact), not causes (CPU spike) — unless cause alerts are the only early warning
• Require 2+ consecutive breaches before firing (avoid flapping)
• Review alert fatigue quarterly: if an alert fires > 10x/month without action, fix or delete it

Step 6: Generate the instrumentation plan

Compile everything into a single document:

Instrumentation Plan — (Project name)

Generated: (date) System: (brief description) Current state: (summary of what's instrumented today)

SLIs, SLOs & Error Budgets

(Table from Step 2)

DORA Metrics Baseline

(Table from Step 3 — current state, targets, measurement sources)

Infrastructure Monitoring

(Table from Step 4 — golden signals per component)

Alerting Strategy

(Tiered alert definitions from Step 5)

Implementation Checklist

Priority-ordered list of what to instrument next:

• (P0) (Most critical gap — e.g., "No availability SLI exists for the primary API")
• (P0) (Second critical gap)
• (P1) (Important but not urgent — e.g., "DORA metrics not tracked; add deployment frequency counter")
• (P1) (Next important item)
• (P2) (Nice to have — e.g., "Add cache hit ratio monitoring for CDN")

Open Questions

(Anything that couldn't be resolved without more information)

Recommended Tools

(Based on the team's stack — only include if the user asked or if there's a clear gap)

Step 7: Review and refine

Ask the user:

• Are the SLOs realistic for your team's current maturity?
• Are any critical components missing from the monitoring table?
• Does the alerting strategy match your on-call setup? (No on-call = no P1 phone alerts.)
• Is the implementation checklist ordered correctly for your priorities?
• Any incidents in the last 6 months that this plan wouldn't have caught?

Adjust based on feedback.

Related skills

• /observability-plan — plan product-level observability for user behavior, events, and task completion
• /security-review — generate attacker stories and security acceptance criteria
• /debug-assist — hypothesis-driven debugging when something goes wrong

Output location

Present the plan as formatted text in the conversation. The user can copy it into their project wiki, engineering docs, or incident response runbook.