1 What Is Impact Evaluation and Why Does It Matter?
Does the opening of a nursery school in a neighborhood increase local employment? Do subsidies aimed at improving workplace safety actually reduce occupational injuries? Do local jurisdiction amalgamations affect voters’ political engagement? How many people died because of the COVID-19 pandemic?
These are all causal questions. They ask whether a specific policy, intervention, or shock produced a change in an outcome of interest. In the language of impact evaluation, we usually refer to this policy, intervention, or shock more generally as the treatment. The treatment may be the opening of a nursery school, the provision of workplace-safety subsidies, a municipal amalgamation, or a large unexpected shock such as the COVID-19 pandemic. The outcome may be parents’ employment, occupational injuries, voter turnout, or pandemic-related deaths.
But how can we answer such questions in a credible way?
1.1 An example: the opening of a nursery school
Consider the first example. Suppose that, at time \(t_0\), a neighborhood has no nursery school. At time \(t_1\), a new nursery school opens. By time \(t_2\), we observe the employment status of parents of young children in that neighborhood.
Sì, ha senso, soprattutto perché le responsabilità di cura ricadono spesso maggiormente sulle madri. Lo inserirei senza appesantire la frase:
Nursery schools may serve several purposes, from supporting child development to helping families manage care responsibilities. In this example, however, we focus on one specific mechanism: access to childcare may free up parents’ time and make it easier for them, especially mothers, to participate in the labor market.
However, observing what happens after the nursery school opens is not enough to establish whether it caused a change in parents’ employment. To answer this question, we need to think in counterfactual terms.
What would have happened to parents’ employment in the treated neighborhood if the nursery school had not opened?
This is the central difficulty of impact evaluation. We observe the employment status of parents in the treated neighborhood after the nursery school opens, but we cannot observe what their employment status would have been, at the same time, if the nursery school had not opened. The counterfactual is therefore missing. The goal of impact evaluation methods is to find a credible way to fill in this missing information.
The causal question is not whether the outcome changed over time, but whether it changed because of the treatment, and by how much.
To answer it, we need a credible counterfactual: a convincing approximation of what the outcome of treated units would have been in the post-treatment period had they not received the treatment.
1.2 Three intuitive comparisons
There are several intuitive ways one might try to estimate the effect of the nursery school.
Option A: We could use a before–after comparison. We could compare parents’ employment in the neighborhood before the nursery school opened, at \(t_0\), with parents’ employment after the opening, at \(t_2\).
Option B: We could use a cross-sectional comparison.1 We could compare parents’ employment in the treated neighborhood after the nursery school opened, at \(t_2\), with parents’ employment in other neighborhoods where no nursery school opened during the same period.
1 The term cross-sectional refers to data collected on different units at a single point in time, providing a snapshot rather than tracking changes over time.
2 The term difference-in-differences refers to a method that compares changes in outcomes over time between treated and untreated units, rather than simply comparing their outcome levels.
Option C: We could use a difference-in-differences comparison.2 We could compare the change in parents’ employment in the treated neighborhood between \(t_0\) and \(t_2\) with the change in parents’ employment over the same period in neighborhoods that did not receive a new nursery school.
Each of these approaches is intuitive. Yet each can also be misleading.
1.3 Why simple comparisons may fail
Option A: A before–after comparison may confuse the effect of the nursery school with other changes occurring over time. Parents’ employment may have increased even without the new facility. For example, the local labor market may already have been on an upward trend before the nursery school opened. In that case, attributing the entire increase in parents’ employment to the nursery school would overstate its effect. Similarly, if a negative economic shock occurred between \(t_1\) and \(t_2\), we might wrongly attribute the consequences of that shock to the nursery school.
Option B: A cross-sectional comparison may also lead to wrong conclusions. Neighborhoods with and without a new nursery school may differ in many ways even before the policy is introduced. They may have different income levels, demographic structures, education levels, labor-market opportunities, or demand for childcare. As these differences are usually related to employment, comparing the two groups of parents after the nursery school opens would not correctly identify the employment effect of the nursery school itself.
Option C: A difference-in-differences comparison is the most sophisticated among these simple alternatives, but it still relies on an assumption that may not hold in this application. The key requirement is the parallel-trends assumption (see Chapter 6): if the nursery school had not opened, parents’ employment in the treated neighborhood would have evolved similarly to parents’ employment in the comparison neighborhoods. However, if parents’ employment was already evolving differently in the treated neighborhood before the opening, this assumption may not be credible. In that case, the estimated effect may still mix the impact of the nursery school with other pre-existing differences in parents’ employment trends.
1.4 Why observational data are challenging
Constructing a credible counterfactual is not easy, especially when working with observational data, as in the examples above. In observational settings, policies are not usually assigned at random. Nursery schools may open in neighborhoods with specific characteristics. Workplace-safety subsidies may be received by firms that are more proactive, more exposed to risk, or better informed about public programs. Municipal amalgamations may involve municipalities with particular administrative or political characteristics.
Sì, lo aggiungerei perché rende l’affermazione più precisa, senza suggerire che l’intera differenza dipenda necessariamente da caratteristiche preesistenti.
For this reason, a simple comparison between treated and untreated units may be misleading: it may reflect, at least in part, pre-existing differences between the two groups rather than solely the causal effect of the treatment.
1.5 Why randomized controlled trials help
Estimating the causal effect would be easier in a randomized controlled trial (RCT). Suppose, for example, that a large number of neighborhoods applied to receive a new nursery school, but budget constraints allowed the government to build only some of them.
If the neighborhoods receiving the new facility were chosen at random, as if by flipping a coin, then, on average, the treated and untreated neighborhoods would be comparable before the policy. This is because random assignment breaks the link between treatment status and the pre-existing characteristics of neighborhoods: richer and poorer areas, neighborhoods with higher or lower employment, and places with stronger or weaker demand for childcare would all have the same probability of receiving the new nursery school.
With a sufficiently large number of neighborhoods, say a few hundred, the law of large numbers helps make the treated and untreated groups similar, on average, in terms of these and all other pre-treatment characteristics, whether observed or unobserved (we will see this in Chapter 4). In that case, the average outcome of neighborhoods without a new facility would provide a credible approximation of the average outcome that treated neighborhoods would have experienced in the absence of the policy.
Randomization is powerful because it creates treated and untreated groups that are comparable on average, by design, before the intervention takes place. Differences in average outcomes observed after the intervention can then be plausibly attributed to the intervention itself, rather than to systematic pre-existing differences between the two groups.
This is why RCTs are widely regarded as the gold standard for causal inference.
1.6 What Can We Do When an RCT Is Not an Option?
In many policy contexts, however, randomization is not feasible. It may be too costly, politically unacceptable, ethically problematic3, or simply impossible because the policy has already been implemented.
3 For example, randomly withholding emergency assistance from some disaster-affected communities would raise serious ethical concerns.
Much of impact evaluation therefore deals with a practical question:
How can we construct a credible counterfactual when random assignment is not available?
The methods discussed in this book offer different ways of answering this question. Each method tries, in its own way, to approximate the missing counterfactual under certain assumptions. The key issue is not to choose the most sophisticated method, but to understand which available method is supported by the most credible assumptions in the empirical setting under analysis.
Donald Rubin puts the point very sharply:
Source: This quote is from Rubin’s interview in Observational Studies (Rubin 2022).“Valid frequentist causal inferences from an observational study can only be obtained if you embed the observational study within a hypothetical randomized experiment. […] Otherwise you are just looking at computer output and implied associations, and making up, generally silly, although sometimes interesting, stories.”
This statement captures a central lesson of impact evaluation: even when we cannot run an RCT, we should still think as if we were trying to approximate one. This means asking what the corresponding RCT would look like. If randomization had been possible, what would the study population have been? Which units would have been randomly assigned to treatment and control? What would the control condition have been?
In observational studies, we must then ask how the actual setting differs from this ideal experiment. Why did some units receive the treatment while others did not? Which pre-treatment characteristics may have influenced treatment assignment? Which untreated units are most comparable to the treated units? And under which assumptions can this observational comparison approximate the comparison that randomization would have created?
This way of thinking is useful because it disciplines the design of observational studies. It forces the researcher to move beyond the mechanical application of an evaluation method and to make explicit the comparison that gives the causal estimate its meaning. In this sense, RCT thinking is not only relevant for randomized experiments. It is also a benchmark for observational studies: it helps clarify the target causal question, the assignment mechanism, the role of untreated units, and the assumptions needed to give an estimate a causal interpretation.
Pearl and Mackenzie (2018) describe causal reasoning as a ladder of causality with three levels.
The first level is association. At this level, we ask whether two variables are related in the observed data. For example, do firms receiving public subsidies have lower workplace accident rates than firms that do not? This is an associational question. We can answer it by comparing accident rates between the two groups or by estimating a regression that relates accident rates to subsidy receipt. However, neither approach, by itself, tells us whether subsidies actually reduce workplace accidents. Firms receiving subsidies may differ from those that do not in ways that also affect workplace safety. A common mistake is to interpret an observed association, perhaps supported by a statistically significant coefficient, as evidence of a causal effect without establishing why that interpretation is justified. The real causal question is not simply whether subsidized firms have fewer accidents, but whether those same firms would have experienced different accident rates without the subsidy. Answering this question requires moving up the ladder.
The second level is intervention. At this level, we ask what would happen if we actively changed something. For example, how would workplace accident rates change if firms were given public subsidies for safety investments? This is the level of causal questions: rather than simply observing what happens, we ask about the consequences of deliberately introducing a treatment or policy.
The third level is counterfactuals. At this level, we ask what would have happened under an alternative state of the world. For example, consider a firm that received a subsidy for safety investments: how many workplace accidents would it have experienced if it had not received the subsidy? This question concerns the same firm under two alternative scenarios: what actually happened and what would have happened otherwise. We observe the first outcome, but the second remains counterfactual.
In this sense, the potential outcomes framework, which we introduce in the next chapter, operates at the counterfactual level of the ladder. It defines causal effects by comparing the outcomes that the same unit would experience with and without treatment, even though only one of these outcomes can be observed.