Working together to teach at the right level
It was sometime in 1999 that Abhijit Banerjee and Esther Duflo from the Massachusetts Institute of Technology’s (MIT’s) Abdul Latif Jameel Poverty Action Lab (J-PAL) visited the cramped Pratham office in Mumbai. First there was scepticism about what economics professors from MIT would find interesting about programmes in Mumbai’s municipal schools. Little did they know that this was the beginning of a journey together, a lifetime of exploring questions, building theories, debunking commonly-held beliefs and developing pathways that combined evidence and action.
In the interactions between the two organizations, J-PAL was always curious to hear what elements Pratham had introduced in its work since the last conversation and why. By the mid-2000s, both sides had the confidence to launch new initiatives on the ground simultaneously with an evaluation. What solidified the relationship was the trust between the two organizations; an openness to try new ideas; and the curiosity to figure out “what could work” to help children learn well. Teaching at the Right-Level (TaRL) is the culmination of a decade of collaborative research, aimed at finding out what works, how it works, and at what scale, to improve learning levels of primary-school-age children. This has been tested inside schools, in communities, by volunteers, by schoolteachers and with the support of cluster school supervisors. These programmes have been evaluated through randomized controlled trials (RCTs), where each intervention has been first tried out in a cohort of schools, randomly chosen from a larger population, and thus any difference between the schools that received the programmes and those that did not, can be attributed to the programme.
In 2005, J-PAL researchers evaluated a Pratham programme in eastern Uttar Pradesh. Here Pratham helped to generate village report cards for schooling and learning—a census done by the village residents. The data was discussed in village meetings and in one arm of the intervention, local village volunteers (unpaid) came forward to teach children who needed extra support. Pratham trained these volunteers, who then worked with groups of children daily after school. The results were encouraging. The likelihood of children who came to these community-based classes being able to read was 25-35 percentage points higher than children in control villages. However, this had no effect on improving teaching-learning practices in the local schools.
The reading camps in Uttar Pradesh led to a series of innovations to test if the same approach could be embedded into the functioning of a regular government school. It was around the same time that Pratham’s annual Annual Status of Education Reports (Asers) were launched. Citizens and governments began to discuss and debate the learning crisis. State governments began to search for ways to improve learning levels in their primary schools. By 2008, state governments, like those in Madhya Pradesh, Bihar, Himachal Pradesh and Punjab, were partnering with Pratham on learning-improvement projects. Pratham named the initiative Read India.
The new intervention/impact evaluation of Read India in 2008 was based on learnings from previous J-PAL evaluations and from the experience that Pratham had accumulated. While Pratham-government partnerships were rolling out in several other states, the impact evaluation was done in Bihar and Uttarakhand. Volunteers continued to be part of the intervention, but this time the main teaching-learning activities were carried out by teachers in school. Teachers and volunteers were trained by Pratham. Teachers selected children who needed remedial support and volunteers worked to give them additional attention. The focus continued to be on building foundational skills (reading and arithmetic). Different combinations of teacher training, monitoring, material and volunteers were tested. An interesting additional component was the summer camps, where each school had two teachers and two volunteers working with children of classes III-V during summer holidays. Children were grouped by their learning level rather than by grade. This was one of the first examples of “teaching at the right level”.
Two school years went by quickly. After the end of the 2010 school year, we went to meet the education minister of Bihar. As we walked in, the minister (a former solicitor general of the state) asked Michael Walton (one of the J-PAL researchers), “Doctor, what’s the diagnosis?” Like a seasoned diplomat, Walton gave the good news first. The one-month summer camp showed good improvement in reading and math. The less heartening news was that during the school year, there was not much progress except where volunteers were involved. The qualitative study done alongside the randomized evaluation indicated that during the school year, teachers felt compelled to complete the curriculum even though many of the children were far below grade level. What all this suggested was that schoolteachers were not to blame; it was the “tyranny of the curriculum” that was the main problem. It also highlighted some important lessons: Pedagogy and teaching effort tailored to the child’s need can have significant impact even with teachers, provided this is the primary focus of their work.
The right to education legislation was passed in 2010. With the new law in place, state governments became preoccupied with infrastructure, inputs and teacher recruitments in order to comply with the norms laid down. Pratham was also busy improving the impact of its instructional programmes, systematically tweaking different elements of the model. Several years passed before governments began considering partnerships again. In 2012-13, J-PAL began discussions with the government of Haryana. At the same time, Pratham was in the process of forging a partnership in Bihar. Both began to think about how the TaRL model could be effectively integrated into government schools.
Banerji and Duflo remember a long walk on the banks of the Charles river on a late summer afternoon in Cambridge in 2012. The summer camps of Bihar had restored Duflo’s faith in teachers. What should a model to be used by government teachers during a normal school day look like? In both states, negotiations with the administration led to 1-2 hours a day being set aside for building foundational skills. It was also decided that they would first work with officials at the cluster level. This cadre of government people was trained by Pratham and then went on to conduct 15-20 days of daily “practice classes” themselves. In the next step, these people trained teachers in schools under them and monitored them. J-PAL quickly put in place an RCT in Haryana. In Bihar, Pratham and the government had their own systems of measurement. Within a few months, it was clear that this strategy was working. J-PAL research in Haryana showed that students in intervention schools scored 0.15 standard deviations higher in the oral reading assessments and 0.14 standard deviations higher on written Hindi exams than students in comparison schools.
Regardless of what students learn in public policy schools or development economics courses, in real life, the path from evidence to action for scaling up is rarely linear. Armed with new evidence from Haryana schools and Pratham’s experience in Bihar, J-PAL and Pratham began to talk to different state governments.
In January 2014, there was a heated discussion in the Bihar chief minister’s office about the possibility of introducing TaRL across the state. Senior bureaucrats of the education department were present, as well as the head of one of the districts that had successfully implemented the programme. Some argued in favour; others suggested that a reform process was needed to build capacity of teachers first. At a critical point in the debate, the seniormost civil servant in the room (who had headed the education department during the earlier evaluation) pointed to the results of the J-PAL study of summer camps that showed the effectiveness of the approach when used by teachers even in a short period of time. This discussion laid the ground for the adoption of some of the key principles of the TaRL approach for learning improvement across Bihar in 2014-15.
In the subsequent months and years, J-PAL’s and Pratham’s efforts led to the transplantation of TaRL in Himachal Pradesh, Jharkhand, Andhra Pradesh and Karnataka. The common elements across these efforts: bottom-up and top-down alignment and understanding of the basic learning goals, first-hand teaching-learning experience of supervisors in running the programme, continuous on-site monitoring and mentoring by supervisors after teacher training, periodic in-depth review for tracking progress in the programme, and setting clear priorities based on baseline and midline data. J-PAL is also beginning a new impact evaluation in Karnataka to test the efficacy of the current model.
The key question for any policymaker, donor or implementer—as they draw on the available evidence—is how to make the most informed decision for replication or scale-up of the evidence, both from the local context and from the global base of impact evaluations that have taken place in other locations. J-PAL endorses a practical framework that policymakers can use to decide whether a particular approach makes sense in their context. The key to the framework is that it breaks down the question, “will this programme work here?”, into a series of questions based on the theory behind a programme. Different types of evidence can then be used to assess the different steps in the generalizability framework. If the general conditions that make a programme work in one place hold elsewhere, then anti-poverty programmes can get traction more widely.
The journey of “what works” to improve children’s learning on scale is an exciting one. While Pratham and J-PAL continue to learn from the efforts in India, together they have recently crossed the Indian Ocean. A new set of initiatives testing the TaRL approach on African soil has just started in Zambia. But we will save that story for another day.
Rukmini Banerji and Shobhini Mukerji are, respectively, CEO of the Pratham Education Foundation and executive director, J-PAL South Asia.
This is the third and final article in a series on the efforts of J-PAL South Asia to promote evidence-informed decision making in India over the past 10 years.
Comments are welcome at firstname.lastname@example.org