App Exploit the potential of natural experiments for policymaking
Nudge enthusiasm for policies shaped by randomized controlled trials towards using naturally-set conditions for policy tests
{{^adFree}}
One fondly-remembered dialogue from the classic British political satire Yes, Prime Minister is by the seasoned civil servant Sir Humphrey Appleby, when he self-assuredly asserts: “Government policy has nothing to do with common sense." Of course, this relates to a bygone era where anecdotes, ideology, heuristics and intuition dictated government policy. The current age is one of rigorous research findings, data, analytics and evaluation of new innovations shaping the public discourse. We call this evidence-based policy- making. It involves compiling robust evidence on what intervention works, monitoring programme delivery and using impact-evaluation to measure its effectiveness, with well-validated learnings serving as inputs to improve schemes, scaling up what’s found to perform well, and diverting funds away from ineffective programmes, among others.
{{^adFree}}
The poster boy of this approach is the exalted ‘randomized controlled trial’ (RCT), which is now being used rather widely for testing the effectiveness of any intervention. Essentially, it takes two time periods (one before the intervention and another after) and two sets of subjects (one on which the intervention is applied, or the ‘treatment group’) and another on which it isn’t (the ‘control group’). Now, by ‘randomizing’ the subjects—which is to say they are almost identical except for their random assignment to a ‘intervention’ or ‘no intervention’ group—such a trial can adjudge the impact of an intervention by observing the desired outcome in the group assigned intervention vis-à-vis the other. In general, if a change observed in the outcome is significantly large in the ‘intervention’ subject group than the other, the RCT would suggest that this difference is on account of the intervention, thereby pointing to its efficacy.
RCTs, though, have their own shortcomings. For instance, Nobel laureate Angus Deaton has argued that randomizing doesn’t actually result in creating identical subjects, so the difference in outcomes need not necessarily be attributed to the intervention alone. Plus, just because an intervention worked in place A doesn’t guarantee it would work in place B (the problem of ‘external validity’). What’s more relevant for policy purposes is that sometimes it is not ethically or politically feasible to choose one subject over another for intervention (imagine choosing one district for vaccination and leaving out another for the simple reason that you want to experiment). In contexts where RCTs are not feasible, other methods have to be explored. This is where natural experiments move from backstage into the spotlight.
{{^adFree}}
Natural experiments occur when a particular intervention has been implemented, but the circumstances around the implementation are not under the control of researchers, unlike in RCTs. Often such interventions occur as part of government programmes or sudden unprecedented events (like an economic crisis or a pandemic). Such events provide unique opportunities to ‘naturally’ divide a sample into treatment and control groups. If good-quality data is available in the pre- and post-intervention periods, strong evidence with limited bias can be found. Often, with an interdisciplinary approach, natural experiments facilitate evidence-based policymaking.
Researchers have exploited geographical units (Saraswat and Bansal utilized the Indo-Gangetic Plains region to derive evidence on the health impact of air pollution in India), pandemics (Fenske, Gupta and Yuan used the influenza pandemic to estimate changes in India’s female labour force participation rate), financial crises (Greenstone, Mas and Nguyen used the Great Depression to understand the effect of credit-market shocks on employment in the US), and interventions (Jensen tested a one-price law in Kerala by exploiting improvements in information through the introduction of mobile phones, while Jensen and Oster studied how the introduction of cable TV led to improvement in Indian school enrolment and reduced domestic violence and son preference by increasing women’s autonomy within the household), among other things.
More specifically, natural experiments have at least three core strengths vis-à-vis RCTs. First, such experiments arising from situations when a policy is being implemented in few states or districts and not in others, or an exogenous event affecting some areas more relative to others, may resolve ethical or political feasibility concerns. These settings may then be utilized for the evaluation of policy impact, as the treatment and control groups were formed without any tinkering by a curious investigator. Second, they provide evidence rooted in real-world conditions, unlike RCTs, which require controlled environments that are often thought to be significant departures from reality. Hence, one can ensure external validity of the results. Third, natural experiments can create timely and robust evidence. Biases brought in by the coincident presence of other factors that influence an intervention’s impact on the outcome and the researchers’ lack of control over policy roll-outs, however, remain matters of concern while using natural experiments.
{{^adFree}}
Natural experiments have been flagged in the past as a potential goldmine for policy insights for India. The Economic Survey of 2016-17 referred to demonetization in 2016 as an interesting natural experiment on the substitutability of cash with other forms of money. The administering of a second dose of a particular covid vaccine after the first dose of another in Siddharthnagar district of Uttar Pradesh this May was another natural experiment for information on the efficacy of a vaccine cocktail in comparison with both doses of a single vaccine. How well India exploits natural test settings for public policy will depend on greater awareness of the concept at all tiers of governance. A national-level effort to crowd-source proposals to study natural experiments would be a leap in the right direction.
Yashaswini Saraswat & Anshuman Kamila are, respectively, assistant director at the development monitoring and evaluation office, Niti Aayog, and assistant director at the economic division, ministry of finance. These are the authors’ personal views.
Milestone Alert!Livemint tops charts as the fastest growing news website in the world 🌏 Click here to know more.
{{^adFree}}
Catch all the Elections News,Business News, Market News, Breaking News Events and Latest News Updates on Live Mint.Download The Mint News App to get Daily Market Updates.