Causal Inference I: Correlation, Causation, and Counterfactuals

So far in this course we have built tools that are very good at answering one kind of question: given what I observe, what should I predict? Clustering, regression, classification, and neural networks all learn associations in data.

But many of the questions we actually care about as data scientists are of a different kind. They are questions about what would happen if we acted:

• If we change the checkout button from blue to green, will more people buy?
• If a patient takes this drug, will they recover faster?
• If a student enrolls in the tutoring program, will their grade go up?
• If the city raises the minimum wage, will employment fall?

These are causal questions, and — as we will see — no amount of predictive accuracy answers them by itself.

A motivating example

A hospital analytics team fits a model on patient records and finds a strong, statistically significant relationship:

Patients who visit the hospital are more likely to die within the year than patients who do not.

Should we conclude that hospitals cause death and recommend people avoid them?

Of course not. Sick people go to hospitals. The hospital visit is associated with death because both are driven by a common cause — being sick.

The model is a perfectly good predictor (“I see a hospital visit, I predict higher mortality”) and a catastrophically bad guide to action (“close the hospitals”). This gap between prediction and intervention is the entire subject of these two lectures. (Angrist and Pischke 2009)

Two different questions

It is worth being precise about the distinction.

Prediction (association)

\[ P(Y \mid X = x) \]

“Among people I observe to have \(X = x\), what is the distribution of \(Y\)?”

This is what supervised learning estimates.

Intervention (causation)

\[ P(Y \mid do(X = x)) \]

“If I set \(X = x\) for everyone, what is the distribution of \(Y\)?”

This is what a decision-maker needs. (Pearl 2009)

In general \(P(Y \mid X=x) \neq P(Y \mid do(X=x))\). The whole game is figuring out when — and how — we can recover the second from data that only shows us the first.

The ladder of causation

Judea Pearl describes three “rungs” of reasoning (Pearl and Mackenzie 2018):

Rung	Question	Example
1. Association	What is?	Seeing — customers who buy diapers also buy beer
2. Intervention	What if I do?	Doing — if I discount diapers, will beer sales rise?
3. Counterfactual	What if I had?	Imagining — would this patient have recovered had they not taken the drug?

Standard machine learning lives almost entirely on Rung 1. Causal inference is about climbing to Rungs 2 and 3.

Correlation is not causation

This slogan is famous, and yet the mistakes it warns about are everywhere.

Consider two variables measured across the summer months:

• Ice cream sales and
• drowning deaths

These are strongly, positively correlated. Does ice cream cause drowning?

No — a third variable, hot weather, drives both. Hot days increase ice cream sales and send more people swimming. Temperature is a confounder.

Confounding, visualized

Once we hold the confounder fixed (look only at days near 80°F), the relationship between ice cream and drowning disappears. There was never a causal link.

Where do spurious associations come from?

A correlation between \(X\) and \(Y\) can arise for several reasons — only one of which is “\(X\) causes \(Y\)”:

• \(X\) causes \(Y\) (the thing we want),
• \(Y\) causes \(X\) (reverse causation),
• a confounder \(Z\) causes both \(X\) and \(Y\) (ice cream / drowning),
• selection / collider bias — conditioning on a common effect (Lecture II),
• pure chance — with enough variables, some will correlate by luck.¹

The potential outcomes framework

To reason precisely, we use the Neyman–Rubin potential outcomes model (Splawa-Neyman et al. 1990; Rubin 1974).

For each unit \(i\) (a patient, user, student) and a binary treatment \(T_i \in \{0,1\}\):

• \(Y_i(1)\) — the outcome we would see if unit \(i\) were treated,
• \(Y_i(0)\) — the outcome we would see if unit \(i\) were not treated.

The individual treatment effect is \[ \tau_i = Y_i(1) - Y_i(0). \]

These are called potential outcomes because, for any given unit, only one of them is ever realized.

The fundamental problem of causal inference

We only ever observe the outcome corresponding to the treatment the unit actually received:

\[ Y_i^{\text{obs}} = T_i\, Y_i(1) + (1 - T_i)\, Y_i(0). \]

The other potential outcome — the counterfactual — is missing.

Unit	\(T_i\)	\(Y_i(0)\)	\(Y_i(1)\)	\(\tau_i\)
Ann	1	?	7	?
Bob	0	4	?	?
Cara	1	?	9	?
Dan	0	3	?	?

We can never compute \(\tau_i\) for a single individual. Causal inference is fundamentally a missing data problem (Imbens and Rubin 2015).

From individuals to averages

Since individual effects are unknowable, we target population averages. The key estimand is the Average Treatment Effect:

\[ \text{ATE} = \mathbb{E}[\,Y(1) - Y(0)\,] = \mathbb{E}[Y(1)] - \mathbb{E}[Y(0)]. \]

Related quantities you will meet:

• ATT — the effect on the treated, \(\mathbb{E}[Y(1) - Y(0)\mid T=1]\),
• CATE — the conditional effect for a subgroup, \(\mathbb{E}[Y(1) - Y(0)\mid X=x]\) (the basis of “heterogeneous treatment effects” and uplift modeling).

Why we can’t just compare treated vs. untreated

The tempting estimator is the naive difference in means:

\[ \underbrace{\mathbb{E}[Y \mid T=1] - \mathbb{E}[Y \mid T=0]}_{\text{what we can measure}}. \]

Add and subtract the missing counterfactual \(\mathbb{E}[Y(0)\mid T=1]\) to decompose it:

\[ \underbrace{\mathbb{E}[Y(1)\mid T{=}1] - \mathbb{E}[Y(0)\mid T{=}0]}_{\text{naive difference}} = \underbrace{\text{ATT}}_{\text{causal}} + \underbrace{\mathbb{E}[Y(0)\mid T{=}1] - \mathbb{E}[Y(0)\mid T{=}0]}_{\text{selection bias}}. \]

The selection bias term is nonzero whenever treated and untreated units would have differed even without treatment — exactly the hospital example. Our job is to make that term vanish.

Confounding in numbers: a kidney stone study

A famous real study compared two treatments for kidney stones (Charig et al. 1986). Here are the recovery rates:

	Treatment A (surgery)	Treatment B (less invasive)
Small stones	81/87 = 93%	234/270 = 87%
Large stones	192/263 = 73%	55/80 = 69%
Overall	273/350 = 78%	289/350 = 83%

Treatment A wins for small stones and for large stones — yet Treatment B wins overall. This reversal is Simpson’s paradox.

What is going on?

# Recovery counts (successes, total) by treatment and stone size
data = {
    "A": {"small": (81, 87),  "large": (192, 263)},
    "B": {"small": (234, 270), "large": (55, 80)},
}

for tx in ("A", "B"):
    s_succ, s_tot = data[tx]["small"]
    l_succ, l_tot = data[tx]["large"]
    overall = (s_succ + l_succ) / (s_tot + l_tot)
    print(f"Treatment {tx}: small={s_succ/s_tot:.0%}, "
          f"large={l_succ/l_tot:.0%}, overall={overall:.0%}")

Treatment A: small=93%, large=73%, overall=78%
Treatment B: small=87%, large=69%, overall=83%

Stone size is a confounder. Doctors gave the risky Treatment A to the hard (large-stone) cases and Treatment B to the easy (small-stone) cases. The overall numbers mix effect with case-mix.

Which number should we trust?

• The within-stratum comparisons hold the confounder (stone size) fixed, so they reflect the treatment effect. Treatment A is better.
• The overall comparison is contaminated by selection bias.

The paradox is not a mathematical curiosity — it is a warning: aggregated associations can point the opposite direction from the truth. Deciding which variables to adjust for is itself a causal question (Lecture II).

A second real example: Berkeley admissions

In 1973, UC Berkeley graduate admissions data showed (Bickel et al. 1975):

• Overall: men were admitted at a noticeably higher rate than women — suggesting bias.
• Within almost every department: women were admitted at an equal or slightly higher rate.

The confounder was department choice: women applied disproportionately to more competitive departments with low admission rates. Same paradox, same lesson.

The gold standard: randomized experiments

How do we break confounding by design? Randomly assign the treatment.

If a coin flip decides \(T_i\), then treatment is statistically independent of the potential outcomes: \[ \{Y_i(0), Y_i(1)\} \perp\!\!\!\perp T_i. \]

Independence kills the selection-bias term, because the treated and untreated groups are — in expectation — identical in every respect (measured and unmeasured) except the treatment: \[ \mathbb{E}[Y(0)\mid T{=}1] = \mathbb{E}[Y(0)\mid T{=}0]. \]

So the naive difference in means becomes an unbiased estimate of the ATE (Imbens and Rubin 2015).

Randomization removes confounding

The true effect is \(+5\). The observational estimate is badly biased (sick patients both got the drug and had worse outcomes). The randomized estimate recovers the truth.

RCTs in the wild

Randomized experiments are everywhere once you look:

• Medicine — randomized clinical trials for drug approval,
• Tech — A/B tests are RCTs for product changes (button colors, ranking algorithms),
• Economics & policy — randomized rollouts of cash transfers, job training,
• Agriculture — R. A. Fisher’s randomized field trials, where it all began.

When you can randomize, do it. But often you cannot — for cost, ethics, or because the “treatment” already happened. That is the subject of Lecture II.

Recap

• Prediction \(\neq\) intervention: \(P(Y\mid X)\) is not \(P(Y\mid do(X))\).
• Correlation can arise from causation, reverse causation, confounding, selection, or chance.
• Potential outcomes \(Y(1), Y(0)\) define the individual effect \(\tau_i\); we can never observe both — the fundamental problem of causal inference.
• We therefore estimate averages like the ATE, and the naive difference in means \(=\) causal effect \(+\) selection bias.
• Confounding produces Simpson’s paradox; adjusting for the right variable can reverse a conclusion.
• Randomization makes treatment independent of potential outcomes, removing bias — the gold standard.

Next: when you cannot randomize, how do causal graphs tell you what to adjust for — and what not to?

References

• Causal Inference: What If — Hernán & Robins (free PDF)
• The Book of Why — Pearl & Mackenzie (popular intro)
• Causal Inference: The Mixtape — Cunningham (free online)
• Mastering ’Metrics — Angrist & Pischke

Bibliography

Angrist, Joshua D., and Jörn-Steffen Pischke. 2009. Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton University Press.

Bickel, P. J., E. A. Hammel, and J. W. O’Connell. 1975. “Sex Bias in Graduate Admissions: Data from Berkeley.” Science 187 (4175): 398–404. https://doi.org/10.1126/science.187.4175.398.

Charig, C. R., D. R. Webb, S. R. Payne, and J. E. Wickham. 1986. “Comparison of Treatment of Renal Calculi by Open Surgery, Percutaneous Nephrolithotomy, and Extracorporeal Shockwave Lithotripsy.” British Medical Journal (Clinical Research Ed.) 292 (6524): 879–82. https://doi.org/10.1136/bmj.292.6524.879.

Imbens, Guido W., and Donald B. Rubin. 2015. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press.

Pearl, Judea. 2009. Causality: Models, Reasoning, and Inference. 2nd ed. Cambridge University Press.

Pearl, Judea, and Dana Mackenzie. 2018. The Book of Why: The New Science of Cause and Effect. Basic Books.

Rubin, Donald B. 1974. “Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies.” Journal of Educational Psychology 66 (5): 688–701. https://doi.org/10.1037/h0037350.

Splawa-Neyman, Jerzy, D. M. Dabrowska, and T. P. Speed. 1990. “On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.” Statistical Science 5 (4): 465–72. https://doi.org/10.1214/ss/1177012031.

Footnotes

1. See Tyler Vigen’s Spurious Correlations (https://tylervigen.com/spurious-correlations) for entertaining examples, e.g. US cheese consumption vs. deaths by bedsheet entanglement.