Replicating Experimental Impact Estimates With Nonexperimental Methods in the Context of Control-Group Noncompliance

Publisher: Statistics and Public Policy (published online ahead of print, subscription required)
Dec 14, 2015
Brian Gill, Joshua Furgeson, Hanley Chiang, Bing-Ru Teh, Joshua Haimson, and Natalya Verbitsky-Savitz
A growing literature on within-study comparisons (WSC) examines whether and in what context nonexperimental methods can successfully replicate the results of randomized experiments. WSCs require that the experimental and nonexperimental methods assess the same causal estimand (Cook, Shadish, and Wong, 2008). But experiments that include noncompliance in treatment assignment produce a divergence in the causal estimands measured by standard approaches: the experiment-based estimate of the impact of treatment (the complier average causal effect, CACE) applies only to compliers, while the non-experimental estimate applies to all subjects receiving treatment, including always-takers.
We develop a new replication approach that solves this problem by using nonexperimental methods to produce an estimate that can be compared to the experimental intent-to-treat (ITT) impact estimate rather than the CACE. We demonstrate the applicability of the method in a WSC of the effects of charter schools on student achievement. In our example, some members of the randomized control group crossed over to treatment by enrolling in the charter schools. We show that several nonexperimental methods that incorporate pre-treatment measures of the outcome of interest can successfully replicate experimental ITT impact estimates when control-group noncompliance (crossover) occurs—even when treatment effects differ for compliers and always takers.