From instincts to evidence: a policymaking journey
Many policymakers are keen to move from policymaking by instinct and ideology to evidence-based methods
Premium
Muttai eppadi irukku (How is the egg)?" I asked the happy, albeit thoroughly confused children at a primary school in Tamil Nadu. It was 1998, and only my second assignment in the Indian Administrative Service (IAS), as a sub-collector in the state that pioneered the midday meal scheme. Scaled up to the entire country, the scheme is credited with having increased school enrolment and attendance.
Gains in school enrolment and attendance had slowed and orders were to fix the problem. A week before my adventure, over lunch at the circuit house, a senior official was lamenting the pressure to show results. As he dug into his biryani, he paused, looked at the rice-covered egg and announced, “It must be the bad quality of eggs at the school meals." And so overnight, many of us in the district administration became “egg inspectors", arriving unannounced at schools to peer into cooking pots, slicing eggs, smelling them and talking to the children.
This would have been just a funny anecdote in the life of a civil servant if it were not for the fact that it illustrates something widespread, with hugely negative consequences for millions: So many important decisions and policies around the world are based on instinct, inertia, ideology or ignorance (four Is) rather than data or rigorous evidence. I now work at J-PAL (Abdul Latif Jameel Poverty Action Lab), a centre started at the economics department of the Massachusetts Institute of Technology (MIT), whose network of 158 professors from 51 universities runs hundreds of field projects in 80 countries. Our mission is to promote the use of scientific evidence to inform policy and reduce poverty. In this role, I have seen first-hand the widespread reliance on these four Is in decision making in non-profits, donors, governments and the private sector.
While the use of evidence and data in decision making leaves much to be desired, J-PAL’s experience in India over the past 10 years, working with the Central and state governments, has been one of positive surprises and hope. There have been disappointments, but it is also a story of many in the civil service and political leadership who were keen to understand existing evidence and design innovative solutions. They were willing to be patient in piloting and evaluating these solutions and incorporate this learning into final policies and programme design.
In the first of this series of articles, Abhijit Banerjee and Esther Duflo talked of how randomized control trials (RCTs) can be a powerful tool to understand the true impact of programmes. Our first large-scale research collaboration with a government was 10 years ago in Rajasthan, with the state police, to test innovations to enhance police performance and improve public opinion. Police stations across the state were randomly chosen to receive different innovations, including training to improve the soft skills and scientific techniques of investigating officers; a freeze on transfers of officials to test the impact of frequent transfers on policing; adding local community observers at police stations to increase public awareness of the roles of the police and improve police behaviour through informal monitoring; and a weekly day off to measure its impact on overworked police.
The study found that having a day off increased staff morale, but did not generate significant changes in perceived police performance. Community observers also had no effect on public perception of the police. In contrast, increasing the number of trained officers led to a twofold increase in victim satisfaction, with little impact on other police activities, such as registering cases, asking for bribes or making arrests. A small reduction in transfers also produced an increase in citizens’ and victims’ satisfaction levels. Many of these interventions are often cited as the most promising. With this one study, the Rajasthan government was able to test the impact (or lack thereof) of all of these. Based on the results, the government, with the support of the Bureau of Police Research and Development, scaled up both investigation and soft skills training across the state.
Another instance of innovative research collaboration was between J-PAL and the Gujarat government in the 2009-14 period, to reduce air and water pollution. J-PAL’s affiliated professors partnered with the Gujarat Pollution Control Board (GPCB) to test the effectiveness of an improved third-party audit system on audit accuracy and pollution. In Ahmedabad and Surat, 233 factories were randomly chosen to receive a new audit system in which auditors were randomly assigned to the plants they would monitor, paid from a common pool, and monitored for accuracy. 240 factories, the comparison group, continued with the status quo system.
The results were striking and found that auditors in the comparison group, hired by client factories, systematically under-reported pollution readings to be just below the regulatory standard. The random audit treatment eliminated this large bias as factories in the treatment group responded to independent audits by significantly reducing their pollution emissions. Based on these results, in 2015, the GPCB developed a new software system that tracks and manages all interactions with factories and randomly assigns auditors so that neither the auditors nor the regulators have discretion regarding who audits a particular factory. The reduction in pollution will benefit millions of residents of Gujarat.
While these examples included designing and testing a new intervention, often policymakers only need timely access to rigorous and context-relevant evidence, and some technical advice to adopt it. In 2010, J-PAL initiated discussions with various Bihar government departments. School-based deworming, which had been rigorously tested by J-PAL affiliates in other contexts, was jointly identified as a promising intervention—not only for its health benefits but also for its positive impact on schooling, given the prevalence of worms in many parts of the state. The Bihar government, with support from J-PAL’s partner organization Deworm the World, launched a deworming campaign in 2011 that reached 17 million children. Other large-scale deworming programmes were implemented in Andhra Pradesh, Delhi and Rajasthan. In 2015, the Centre launched a national deworming programme and announced that by October it had treated 89 million children—a win for evidence-based policymaking.
One of J-PAL’s most extensive partnerships has been with the Tamil Nadu government. Launched in 2013, it began with a months-long policy research dialogue that brought together a dozen professors and researchers from J-PAL, and secretaries of various departments like education, health, labour, civil supplies, rural and urban development. They discussed key priorities of the state and the latest scientific evidence from around the world on these issues. This led to the design of innovative pilots that address the challenges, and evaluations to measure their impact. This partnership has led to more than a dozen projects involving eight state departments and 24 researchers to inform policy. In addition, J-PAL is also providing training and advice on creating a data analytics unit in the planning department for better collection and timely analysis of data for policymakers.
A similar broad partnership is under way with the Punjab government. In neighbouring Haryana, J-PAL is working with the health department to pilot innovative technology and incentives to increase immunization. And J-PAL researchers working with the Andhra Pradesh government conducted one of the largest field studies on reducing leakages in the Mahatma Gandhi National Rural Employment Guarantee Scheme. J-PAL South Asia has 38 similar ongoing or completed projects with 16 state governments and four with the Central government. We have also partnered with government agencies to help strengthen the capacity of civil servants to use evidence in policymaking. This includes modules for IAS officers at the National Academy of Administration since 2011, and for the Indian Economic Service.
Every engagement between a government and an organization like J-PAL, committed to generating and disseminating rigorous scientific evidence, helps both sides better understand how policymakers and researchers can partner to design, evaluate and scale up innovative solutions to reduce poverty and accelerate development. In the last decade, we have learnt how hard it can be to align the needs and constraints of policymakers and researchers, and yet how beneficial such partnerships can be. We have seen how so many in the government are open to using evidence to guide their policies but often find it hard to access or understand the relevant evidence. We have understood how even a single civil servant or minister can encourage a culture of evidence-informed policy by committing their time, persuading their colleagues and, most importantly, by forsaking expediency and waiting for results to “get it right". And we have experienced the thrill of seeing millions of people benefit when rigorous research and evidence combine with the government’s extensive field knowledge and reach.
Our experience of working with governments in India was a big inspiration for starting J-PAL’s Government Partnership Initiative, which supports long-term collaborations between researchers and policymakers to institutionalize a culture of evidence-informed policy. As J-PAL South Asia completes 10 years, we are incredibly grateful to all the governments and policymakers who have been at the centre of this journey from instincts to evidence.
As for the instincts of the senior official who ordered those egg inspections, the happy but confused schoolchildren told me (and my colleagues) almost everywhere I went: “Muttai nalla irukku (The egg is good)!"
This is the second in a series of three articles tracing the efforts of J-PAL South Asia to promote evidence-informed decision making in India since its inception 10 years ago. In the first article published in October, J-PAL co-founders Abhijit Banerjee and Esther Duflo talked about the motivation for starting J-PAL’s regional office in India; and in the third article in December, J-PAL South Asia’s executive director Shobhini Mukerji and Pratham’s CEO Rukmini Banerjee will dive deeper into a specific case of using rigorous evidence to understand the drivers of learning outcomes in schools and scale up programmes based on that research.
Iqbal Singh Dhaliwal is the deputy executive director of the Abdul Latif Jameel Poverty Action Lab (J-PAL) at the economics department of the Massachusetts Institute of Technology and the co-scientific director of J-PAL South Asia at IFMR.
Comments are welcome at views@livemint.com
"Exciting news! Mint is now on WhatsApp Channels 🚀 Subscribe today by clicking the link and stay updated with the latest financial insights!" Click here!
