Causal Inference Guide

Use this when you need to answer "did X cause Y?" and can't run a randomized experiment. Common situations: evaluating a feature that already launched without a holdout, measuring the impact of a policy change, understanding whether a marketing campaign drove conversions or just correlated with a seasonal trend. Produces a causal inference design with method selection, assumptions audit, and analysis plan.

Related skills: Complements /experiment-design (use that when randomization is possible; use this when it isn't). Upstream: /north-star-metric identifies the outcome to measure. Downstream: findings feed into /decision-brief for stakeholder communication. Pair with /data-quality-assessment to verify data before analysis.

Process

Step 1: Gather inputs

Ask the user to provide:

1. The causal question -- what treatment or intervention are you trying to measure the effect of? What outcome do you care about?
2. Why randomization failed -- why can't you run (or didn't you run) an A/B test? (Already launched, ethical constraints, too few units, management decision, retroactive analysis.)
3. Treatment assignment -- how were users/units assigned to the treatment? (Self-selected, rule-based cutoff, geographic rollout, time-based rollout, management decision.)
4. Available data -- what data do you have on the treatment group, control group, and outcome? How far back does it go? What covariates are available?
5. Confounders -- what other factors might explain the outcome besides the treatment? (Seasonality, concurrent launches, user self-selection, external events.)
6. Decision context -- what will you do with the answer? How precise does the estimate need to be?

Step 2: Select the causal method

Match the situation to the appropriate method:

## Causal Inference Design -- {{question}}, {{date}}

### Method selection

| Method | When to use | Key assumption | Your situation |
|--------|------------|----------------|---------------|
| Difference-in-Differences (DiD) | Treatment applied to one group at a known time; parallel pre-treatment trends available | Parallel trends: treated and control would have followed the same trajectory absent treatment | (Fit? Why or why not?) |
| Regression Discontinuity (RDD) | Treatment assigned by a threshold/cutoff (e.g., users above a score get the feature) | Continuity: units just above and below the cutoff are comparable | (Fit? Why or why not?) |
| Propensity Score Matching (PSM) | Treatment is self-selected but you have rich covariates to model selection | No unmeasured confounders: all variables driving selection are observed and included | (Fit? Why or why not?) |
| Instrumental Variables (IV) | A variable affects treatment assignment but not the outcome directly | Exclusion restriction: the instrument only affects the outcome through the treatment | (Fit? Why or why not?) |
| Interrupted Time Series (ITS) | Single group, treatment at a known time, long pre/post time series | No concurrent events: nothing else changed at the same time that could explain the shift | (Fit? Why or why not?) |
| Synthetic Control | One treated unit (e.g., a market or country), multiple untreated units for comparison | Pre-treatment fit: the synthetic control accurately reproduces the treated unit's pre-treatment trajectory | (Fit? Why or why not?) |

### Selected method: {{method_name}}

**Rationale:** (Why this method fits the situation better than alternatives.)

**Key risk:** (The biggest threat to validity with this method in this context.)

Step 3: Audit assumptions

Every causal method requires assumptions. Make them explicit and testable where possible:

### Assumptions audit

| Assumption | Required by | Testable? | Test or evidence | Status |
|-----------|------------|-----------|-----------------|--------|
| (e.g., Parallel trends) | DiD | Yes | Plot pre-treatment trends for treated vs. control; run placebo test on pre-period | (Pass / Fail / Untested) |
| (e.g., No spillover) | DiD | Partially | Check if control group behavior changed after treatment started | (Pass / Fail / Untested) |
| (e.g., No anticipation) | DiD | Partially | Check for behavior changes before the treatment date | (Pass / Fail / Untested) |
| (e.g., Overlap/common support) | PSM | Yes | Plot propensity score distributions; check for non-overlapping regions | (Pass / Fail / Untested) |

### Threats to validity
1. (Specific threat -- e.g., "Marketing campaign launched the same week as the feature, confounding the treatment effect")
2. (Specific threat -- e.g., "Power users self-selected into the beta, making treated users systematically different")
3. (Specific threat -- e.g., "Only 3 months of pre-treatment data, making parallel trends hard to verify")

### Mitigation strategies
- (For each threat: what can you do to reduce the risk or bound the bias?)

Step 4: Design the analysis plan

### Analysis plan

**Treatment definition:**
- Treatment group: (Who/what is treated? How are they identified in the data?)
- Control group: (Who/what is the comparison? How are they identified?)
- Treatment timing: (When did the treatment start? Is it binary or staggered?)

**Outcome variable:**
- Primary: (The main metric you're measuring the effect on)
- Secondary: (Other metrics to check for consistency or mechanism)

**Covariates:**
- (Variables to control for or match on -- list all that are available and relevant)

**Estimation approach:**
- (Specific regression specification, matching algorithm, or time series model)
- (Software/tool: R/Python package, SQL approach)

**Effect size and precision:**
- Expected effect size: (What change would be meaningful? What's the minimum you need to detect?)
- Standard errors: (Clustered? Heteroscedasticity-robust? Bootstrap?)
- Confidence intervals: (Report 95% CI, not just p-values)

**Robustness checks:**
1. (e.g., Placebo test: run the same analysis on a pre-treatment period where no effect should exist)
2. (e.g., Sensitivity analysis: how large would an unmeasured confounder need to be to explain away the result?)
3. (e.g., Alternative control group: repeat with a different comparison group)
4. (e.g., Dose-response: if treatment intensity varies, does the effect scale with exposure?)

Step 5: Plan the communication

### Communicating causal findings

**For technical audiences:**
- Report the point estimate with 95% confidence interval
- State the method and key assumptions explicitly
- Present robustness checks alongside the main result
- Quantify what would need to be true for the result to be wrong (sensitivity analysis)

**For non-technical stakeholders:**
- Lead with the business question and the answer
- State the confidence level in plain language: "We're reasonably confident that X caused a Y% increase in Z, with the caveat that [main limitation]"
- Use one visual: before/after trend with treatment marked, or treated vs. control comparison
- Be explicit about what this analysis can and cannot tell you

Step 6: Review

Ask the user:

• Is the treatment cleanly defined? (If users could partially adopt or if treatment leaked to the control group, estimates will be biased.)
• Is the control group a fair comparison? (If treated users are systematically different, the method needs to address selection.)
• What would change your mind? (If no robustness check could shake your confidence, you're not being rigorous enough.)
• Who needs to trust this analysis? (A data scientist will want robustness checks; an executive wants a clear bottom line.)
• Is a directional answer sufficient, or do you need a precise estimate? (This determines how much methodological investment is warranted.)

Output location

Present the causal inference design as formatted text in the conversation. The analysis plan can be exported for the data team to execute.