The Backbone of Impact Evaluation Methods
Back to Basics
About the book
This book is not intended to be a comprehensive textbook on impact evaluation and causal inference. Many excellent books and articles already provide rigorous and detailed accounts of causal inference, program evaluation, and econometric methods for policy analysis.
The purpose of this book is different. It is to go back to basics and offer a simple and transparent perspective on the logic behind the most widely used impact evaluation methods. Rather than emphasizing the technical aspects of evaluation methods, the book aims to clarify how each method constructs a counterfactual, what assumptions it requires, and how it works in practice, while keeping formal details to a minimum.
In this introduction, I will mention some methods and concepts that may be unfamiliar to readers new to causal inference. Do not worry: each of them will be covered in the book, assuming only very basic statistical knowledge on the part of the reader. If you are completely unfamiliar with these topics, you may prefer to skip this introduction on a first reading and start directly with Chapter 1, returning to it later once the main concepts have become more familiar.
How to estimate treatment effects?
The central question of ex-post impact evaluation is simple: What would have happened to the outcome of interest in the absence of the treatment? For instance, what would have happened to students’ test scores if schools had not introduced a new tutoring program? Or what would have happened to firms’ investment if they had not received a public subsidy?
The focus is usually on the units exposed to the treatment, such as students or firms. For these units, we observe the post-treatment outcome—for example, their test scores or investment levels—but not what would have happened had they remained untreated. The central challenge is therefore to estimate this missing counterfactual. Once it is recovered, the treatment effect is obtained by comparing the observed average post-treatment outcome with its estimated counterfactual value.
Crucially, for any given unit at a given point in time, we can observe only one of its potential outcomes, while the counterfactual outcome remains unobserved. To estimate treatment effects credibly, we therefore need to construct a plausible approximation of it. How can this be done? The most common approach1 is to exploit information from untreated units. The basic idea is that untreated units can help approximate what would have happened to treated units in the absence of the treatment.
1 Though not the only one, as discussed in Chapter 11
We will see how the most widely used evaluation strategies within this family of control-based counterfactual methods (CBCMs)—randomized controlled trials (RCTs), matching methods, difference-in-differences (DiD), instrumental variables (IV), regression discontinuity (RD), and the synthetic control method (SCM)—exploit information from untreated units to retrieve the treatment effects of interest.
In this book, a central theme is transparency. For this reason, I will show how each method constructs its counterfactual, which units are used to build it, and under which assumptions.
Transparency as the Guiding Principle of This Book
In recent years, transparency has become central to the debate surrounding machine and deep learning algorithms. Complex predictive models are often difficult to interpret, and it is not always clear how an algorithm transforms data into predictions. For this reason, researchers and practitioners increasingly ask for models that are not only accurate, but also interpretable and transparent. At the same time, much less attention is usually paid to transparency in the context of impact evaluation methods. It is often taken for granted that causal methods are transparent by nature. Yet, as discussed below, this is not necessarily true for some of the most commonly used approaches.
In a successful RCT, the logic of the comparison is usually easy to understand. Random assignment makes treated and untreated units comparable before the intervention: on average, the two groups are similar in all respects except for the treatment they receive. For example, in a trial testing a new drug, patients are randomly assigned to one of two groups: a treatment group that receives the drug and a control group that does not, so differences in health outcomes after treatment can be credibly attributed to the drug itself. As a result, the untreated group provides a natural benchmark for what would have happened to the treated group in the absence of the intervention. Similarly, in a sharp RD design (see Chapter 9), the role of the comparison units is relatively intuitive. Units just below and just above the cutoff are compared because they are expected to be very similar in all respects, except for treatment assignment.
This clarity is often less immediate in other widely used methods. For instance, in a matching application (see Chapter 5), it is not always disclosed which untreated units are actually used to construct the counterfactual, how much weight each of them receives, and whether all treated units are retained in the final matched sample. In a staggered DiD design (see Chapter 7), describing the comparison group simply as “untreated” can obscure which units actually serve as controls, whether not-yet-treated units are included, and how much the estimate depends on particular groups or time periods. The same issue becomes even more complex in IV designs (see Chapter 8), especially when the instrument is non-binary, when multiple instruments are used, or when the empirical specification includes several fixed effects, covariates, and trends.
Impact evaluation methods may look deceptively simple at first sight. In practice, however, understanding how they construct the counterfactual is often less straightforward, because this process is not always transparent.
To interpret the estimates causally, we need to understand which units are being compared and what assumptions are required to identify the causal effects of interest.
The lesson from SCM. One reason SCM has been so influential over the past two decades is that it makes it easy to see how the counterfactual is constructed from the data (see Chapter 10). SCM builds the counterfactual by choosing a weighted combination of untreated units through an optimization algorithm that seeks to reproduce the treated unit’s pre-treatment outcomes and characteristics as closely as possible, although the underlying procedure may not be immediately intuitive. However, once the optimization process is completed, the method reports the weights assigned to each control unit and usually presents a graph comparing the treated unit with its synthetic counterpart. This makes it possible to see which untreated units contributed to the counterfactual, how much they contributed, and whether the synthetic control provides a credible pre-treatment fit—a key condition for considering the estimated treatment effect as reliable.
This feature is not only useful from a methodological point of view; it is also pedagogically powerful. It reminds us that control-based impact evaluation methods ultimately rely on comparisons between treated and untreated units. They are not magic formulas that automatically identify the causal effect of interest; they are structured ways of making comparisons credible under specific assumptions.
No ground truth. This point is particularly important because in observational studies, we do not observe a ground truth causal effect. We observe what happened after a policy was introduced, but we do not observe what would have happened to the same units, at the same time, in the absence of that policy. This makes it difficult to compare evaluation strategies as we often compare machine learning algorithms for prediction. In prediction problems, models can be assessed against observed outcomes, and their accuracy can be compared directly. For example, algorithms predicting the sale prices of flats can later be evaluated by comparing their forecasts with the actual prices at which those flats were sold. In impact evaluation, instead, the counterfactual outcome is always missing. We therefore cannot simply run a “horse race” among methods and declare which one is empirically more accurate. For this reason, the credibility of an impact evaluation strategy depends mainly on the plausibility of its identifying assumptions and on the transparency of the comparison it constructs.
Why causality matters
Many important questions in economics, policy evaluation, business, political science, and many other fields are causal questions. We do not only want to know whether two variables move together. We want to know whether changing one of them would actually change the other. This is the idea behind the familiar motto: association is not causation. Two variables may be strongly associated because one causes the other, but also because they are both driven by a third factor, because of selection, or simply because the observed relationship reflects a misleading comparison. Causal inference starts from this distinction.
Would a conditional cash transfer increase children’s school attendance? Would a public subsidy reduce workplace accidents? Would smaller classes improve student achievement? Would a carbon tax reduce firms’ energy consumption? To answer these questions, simply comparing treated and untreated units in the post-treatment period is not enough. We need to understand what would have happened to the treated units during that same period if they had not received the treatment.
Estimating causal effects is difficult because this counterfactual outcome is not directly observed. We observe what happened after the policy change but not what would have happened otherwise. Impact evaluation methods are ways of making this missing comparison as credible as possible.
This is why causality is at the center of empirical work. Without a credible causal design, data can reveal patterns, correlations, and associations, but it cannot by itself tell us what works, for whom, and under which conditions.
A causal effect is defined with respect to a specific intervention, outcome, target population, and counterfactual state of the world. For this reason, causal analysis should usually begin by clarifying one causal question at a time.
For example, suppose the outcome of interest is workers’ earnings. Asking whether a training program increases earnings is not the same as asking whether a wage subsidy, job-search assistance, or access to childcare increases earnings. The outcome is the same, but each question involves a distinct treatment, counterfactual comparison, and possibly target population. Mixing these questions in a single analysis can make it unclear which causal effect is being estimated.
This matters because credible causal inference requires a precise answer to the question: compared to what? The same observed outcome may be consistent with many different causal stories. To estimate an effect, we need to define the treatment, the untreated state, the units of interest, the timing of the intervention, and the outcome to be evaluated.
Trying to estimate several causal effects at once often leads to ambiguity. It becomes difficult to understand which counterfactual is being constructed, which assumptions justify it, and what the estimated coefficient actually captures. A clear impact evaluation therefore starts by isolating one causal question, defining its counterfactual carefully, and then choosing an empirical strategy that can make that comparison credible.
The rest of the book
After covering the basic concepts of impact evaluation and the most widely used evaluation approaches, the second part of the book turns to selected topics in impact evaluation, including:
Forecasting-based counterfactual methods (FBCMs). These methods do not build the counterfactual by comparing treated units with untreated units. Instead, they use the treated unit’s pre-treatment trajectory and other predictors to forecast what its outcome would have been in the post-treatment period without the intervention. The estimated treatment effect is the difference between the observed post-treatment outcome and this forecasted counterfactual.
Spillovers and interference. We will discuss what happens when the treatment affects not only treated units, but also the outcomes of some or all untreated units. In these settings, untreated units may no longer provide a ‘clean’ counterfactual.
Continuous treatments. Throughout most of the book, we focus on binary treatments, in which each unit either receives the treatment or does not. This setup makes the counterfactual logic especially clear: the goal is to compare outcomes under treatment with outcomes in the absence of treatment. In this chapter we will consider more complex settings in which the treatment varies in intensity. In this case, the causal question is not only whether the treatment has an effect, but how the outcome changes as treatment intensity increases.
How to implement impact evaluation methods in practice. Through simulated scenarios, I will show step by step how the different methods can be implemented in practice. Replication code will be provided in R.
Impact evaluation is not primarily about applying sophisticated estimators2. It is about constructing a credible counterfactual scenario.
Different methods provide different ways of approximating the missing counterfactual, and each of them relies on specific assumptions that should be assessed case by case, depending on the empirical setting, the available data, and the institutional context.
2 An estimator is a rule or statistical procedure used to estimate an unknown quantity from observed data. For example, the sample mean is an estimator of the population mean.
The book therefore does not try to cover every method, every extension, or every recent technical debate. Instead, it focuses on the basic logic of the main approaches and on the practical questions that arise when these methods are used to evaluate public policies or other treatments.
The audience of this book
The book is written for readers who want to understand the intuition behind impact evaluation methods before moving to technical details. It is intended for students, applied researchers, policy analysts, and practitioners who use causal methods or read studies based on them.
Formal notation is kept to a minimum. The emphasis is on interpretation, design, credibility, and transparency.
Readers who feel less comfortable with formal notation may first consult the Appendix, which offers a basic guide to essential concepts in statistics, econometrics, and causal inference.
Acknowledgments
Before taking the first steps in this journey, I would like to thank my mentor, Professor Guido Pellegrini, who taught me much of what I know about public policy evaluation. I am also grateful to my co-authors: through many discussions with them, I have refined my understanding of the discipline. Finally, I wish to thank the many curious students and public administration practitioners whose questions and insights have encouraged me to explain key concepts more clearly.
Disclaimer
This is version 0.3 of the book. It is still a work in progress, and I would be grateful for any comments, suggestions, or corrections that could help improve it.
All illustrations in this book were created with the assistance of a generative AI system, based on specific instructions and, in some cases, hand-drawn sketches provided by the author. They are intended for illustrative and educational purposes only.
Except where otherwise noted, © 2026 Augusto Cerqua.
This book is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.