The pitfalls of random control tests for policy purposes4 min read . Updated: 10 Aug 2020, 09:44 PM IST
Such trials may have hogged the limelight in recent times but they’re often neither ethical nor useful
A recent paper published by the National Bureau of Economic Research of Boston created a huge controversy. Titled Enforcing Payment for Water and Sanitization Services in Nairobi’s Slums, it has four co-authors, some of whom are senior researchers with the World Bank. It tackles a fairly typical question faced by policymakers. How does one ensure that public utility services like water, sewerage and electricity achieve full coverage in a sustainable way? At least some user charges have to be collected to cover the cost of delivery. If a large portion of the bills remain unpaid, it can lead to a fiscal crisis, or to rationing of the services, defeating the very purpose of achieving universal coverage. The paper explored what should be the right policy to ensure higher bill collection. Should it involve “soft" persuasion, face-to-face meetings with stakeholders, or a “hard" system of disconnection on non-payment? Researchers designed an experiment that was conducted in the poor neighbourhoods of Nairobi. They divided the sample households into those who would be offered the “soft" option and those who would face “hard" disconnection. The experiment ran for nine months. There were thus slum dwellers who faced water disconnection nine months longer than the other group.
This caused a furore over the ethics of such a randomized control test (RCT) among economists and social activists. How can you justify a test that cuts off the water supply of a control group?
Was there informed consent of participation in this experiment? Can the poor and vulnerable be put to such “inhuman" tests? What about the impact on child health? The research found that “hard disconnection" worked better than “soft" persuasion in the recovery of bills. More troubling was the finding that the political “cost" of hard disconnection is low. In many developing countries, one of the reasons that water and electricity disconnections are not done is the fear of community protests and a political backlash. Often, there is political interference too, as evident in the case of India’s electricity distribution companies.
This research, supported by the World Bank, helped confirm, at least in the Nairobi context, that governments need not be fearful of losing political support. Did the RCT not implicitly provide support to the ruling government? What if the government had otherwise repressive policies in other spheres? Does such an alignment with World Bank researchers help consolidate the legitimacy of repressive regimes?
This RCT-based research paper and its implied policy advice has highlighted the many pitfalls of using such experiments in deciding crucial policy designs. RCTs were in the limelight recently after a Nobel Prize was awarded to three of the methodology’s pioneers, Abhijit Banerjee, Esther Duflo and Michael Kremer. Their work has been quite rigorous, and they have always been careful not to exaggerate the validity of their findings. But not every RCT researcher is like that. Indeed, an RCT “epidemic" is now widespread, infecting donor agencies like the World Bank and Bill & Melinda Gates Foundation, and inevitably influencing policy in the developing world. The dangers of misapplication cannot be overemphasized. In another well-known paper published in 2013 in the Quarterly Journal of Economics, its authors had used the RCT technique for determining how to reduce corruption in the grant of driving licences in India. In their experiment, one control group was “encouraged" to get their licences by paying bribes. The ethics of those fraudulently licensed drivers on Indian roads boggles the mind, and we can’t blame the researchers for the malaise. But the methodology is on ethically slippery ground.
No wonder that critics of RCT are increasing. One of the most prominent ones is Nobel laureate Angus Deaton, who has himself used the RCT methodology on occasion. In his essay in a forthcoming book titled Randomized Controlled Trials in the Field of Development: A Critical Perspective, he writes that although RCTs have been used for a long time, they have no unique advantage or disadvantage over other empirical methods. They do not simplify inference, nor can they establish causality. He makes a scathing remark about the ethics of RCTs done by Western economists on the poor in India. These RCTs were vetted by American institutional review boards, but would not have been allowed on American subjects. In Deaton’s words, “There is an uncomfortable parallel here with the debates about pharmaceutical companies testing drugs in Africa."
The RCT movement gained acceptance and popularity because of the notion that policy should be evidence-based. There was an increasing distrust of deductive reasoning or theoretical models. That also explains the current fad of throwing big data at any policy puzzle. No prior model needed. RCT also claims to be a priori agnostic, and only seeks empirical confirmation through an experiment. But just because it works in the context of a slum in Nairobi, it does not mean that it can be applied everywhere else. And, as Deaton points out, even the limited applicability in the context of one experiment can be questioned. Even if RCTs are useful, that does not mean other modes of study, such as surveys, observations, questionnaires and trial and error, are irrelevant. Not to forget the importance of good old common sense, logic and reasoning. Indeed, an RCT often just ends up confirming common sense.
Ajit Ranade is an economist and a senior fellow at The Takshashila Institution, an independent centre for research and education in public policy.