Friday 26th of November

This week, Anne will present Does the quest for randomization make us lose sight of interesting research questions?

Does the quest for randomization make us lose sight of interesting research questions?

Randomized control trials are often criticized for low external validity, identifying only point estimates for a heterogeneous population, and not allowing us to understand the mechanisms underlying the effect of an intervention. Another concern, less often discussed, is the fact that the quest for randomization also puts restrictions on the questions that we can answer. Conducting an RCT requires a set of well-defined, measurable outcome indicators. This has made research on health and education particularly popular among development economists. Other interesting topics, such as political accountability or the access to justice, are less suitable for randomization, as outcomes in theses fields are more difficult to measure. As RCTs are conducted in real life time, they also prevent us from answering questions about the long-term change of cultural values, or the inter-generational transmission of ideas. Given that an RCT is a large investment in time and resources, randomistas are also unlikely to experiment with interventions whose effect is uncertain. At the same time, the “new generation RCTs” now use more and more sophisticated outcome indicators and are designed to disentangle different impact channels and understand their interaction (see links below for examples).

Some of the questions we can discuss tonight are “Does the quest for randomization prevent us from answering the most interesting research questions?” “Or does randomization instead guide us to the most interesting questions, as it forces us to think about interventions with strong potential for impact?” “Is randomization in this sense any different from other empirical methods, where the availability of data and the existence of an identification strategy also restrict the questions that can be answered?” “How can we ensure that the methodological concerns do not prevent us from answering the most interesting research questions?”

http://www.annualreviews.org/doi/full/10.1146/annurev.polisci.12.060107.155922

http://aidwatchers.com/2009/07/development-experiments-ethical-feasible-useful/

http://aidwatchers.com/2010/07/is-impact-measurement-a-dead-end/