India’s toilet stats raise a stink

India’s villages are now officially open defecation free, according to those who were assigned this target: the sanitation department, now subsumed under the Jal Shakti ministry. But new evidence from the government’s own National Sample Survey (NSS) appear to contradict that claim. The sanitation department has contested the NSS findings but statisticians and independent experts aren’t convinced with the department’s answers.

The 76th NSS round conducted by the National Statistical Organization (NSO) found that as of September 2018, 28.7% of India’s rural households lacked access to toilets and 32% practised open defecation. At that time, official data from the Swachh Bharat Mission (SBM) painted a much more optimistic picture claiming that just 6% of India’s households lacked access to a toilet. And though the NSO has qualified their findings, other external surveys reveal similar discrepancies with administrative data. While SBM has improved sanitation, India may be far from completely open-defecation free, those surveys and independent researchers suggest.

The results of the NSS survey published a few weeks earlier showed that Odisha (49.3% of rural households with access to a toilet), Uttar Pradesh (52%) and Jharkhand (58%) were the worst states in terms of sanitation coverage in the July-December 2018 period. But according to data on the SBM dashboard at that time, sanitation coverage in these states was significantly higher. In Uttar Pradesh, for instance, 99% of rural households had access to a toilet in September 2018, the dashboard had shown.

According to the government, these discrepancies are explained by respondent bias. Both the NSS in its survey report and the Jal Shakti ministry have suggested that respondents may have lied about access to toilets in the hope of availing benefits under SBM. “Respondent bias, in this case, is when individual households, when asked a leading question on whether they have ever received benefits from the government, do not admit they have toilets or LPG cylinders in the hope of receiving additional benefits from the government. In case of the 76th round of the NSS, the question on benefits received preceded that of access, thus leading to a significant under-reporting of sanitation coverage,” a spokesman for the sanitation department wrote in a response to Mint’s queries about the discrepancies between the SBM data and NSS.

Statistical experts, including those who held senior positions in the government earlier, said that the respondent bias cannot explain such large discrepancies, especially when there is also a social desirability bias to over-report access to toilets.

“Not having a latrine would not be seen as a socially and politically desirable position among respondents,” said Pronab Sen, India’s former chief statistician. “On the other hand, not having a latrine is desirable from an economic point of view, if one thinks they are going to get a subsidy. So the question as to which direction the bias exist depends much on how the respondent is reading the purpose of the survey. And, it depends on how the field investigator presented himself and how the tone was set.”

It is difficult to conceal information on a large benefit like a toilet, especially since it can be physically verified, said P.C. Mohanan, a former member of the National Statistical Commission (NSC). Besides, a large number of interviews are conducted in the presence of others, Mohanan added.

Interestingly, though the NSS survey report adds respondent bias as a caveat to the data on access to toilets, it does not add it for other sanitation-related amenities (e.g. taps). The publication of the NSS survey report was delayed because of objections raised by the sanitation department, Business Standard reported earlier. NSO officials did not respond to Mint’s queries on these issues.

This is not the first time administrative data on toilets constructed under SBM have come under a cloud. Independent surveys by organizations such as the Centre for Policy Research (CPR) had shown earlier that the ‘open defecation free’ tag often betrays the ground reality, and portrays latrine ownership rather than usage. Given that those in charge of meeting targets are also reporting these statistics, this creates a reporting bias in administrative data, CPR’s president, wrote Yamini Aiyar in a Hindustan Times article.

Other external surveys also suggest that official data on toilets may be overreported. Data from the Research Institute of Compassionate Economics (RICE), a non-profit research organization, from a survey conducted between September to December 2018 shows access to toilets in rural Bihar, Madhya Pradesh, Rajasthan and Uttar Pradesh was significantly lower than SBM figures at the same time.

This discrepancy could have increased over time as SBM ramped up. For instance an earlier unrelated, large-scale survey on teenage girls, conducted by the NGO Naandi Foundation, found that 55.2% of Indian rural households had access to a toilet in November 2016 largely in line with SBM data at that time (57.1%).

The government has consistently debunked external findings, such as the RICE survey, and cited the results from the National Annual Rural Sanitation Survey (NARSS) conducted by the sanitation department, and funded by the World Bank. This survey found 93% of India’s households have toilets as of February 2019 and is largely in line with SBM dashboard data. The survey’s methods, though, have been questioned by researchers from RICE who suggest that surveyors were pressured to ensure survey data matched administrative data .

Discrepancies in the data also extend to how often toilets are used. For instance, according to NARSS, almost all households (97%) who have a toilet use it. Similarly the NSS survey found that 95.2% of rural households with a toilet use it regularly. But other surveys suggest that the proportion may be lower, with significant variation in usage within households. For instance, a study by IDinsight, an international research organization, found that in Darbhanga district in Bihar, at least one person in a quarter of households with access to toilets defecate in the open, as of January 2019. Yet current SBM data suggests that 100% of the villages in that district are open-defecation free.

All these inconsistencies partly reflect the difficulties in measuring open defecation and the differences in measurement methods. Sanitation is a sensitive issue and social desirability bias could be a significant issue especially amid a national campaign to eliminate open defecation according to 3ie, a global organization focusing on impact evaluation of public policies. In areas declared open defecation free, respondents may feel uncomfortable admitting they practice open defecation . Other issues lie in the definitions of open defecation. The SBM defines it as 100% toilet usage at the household level but the NSS considers a threshold of 50%. Even timing of surveys may matter. For instance, lack of water during summers could mean houses use latrine less during the season and thereby create a bias in reported numbers.

Yet for all the differences in the data, almost all studies suggest that SBM has improved sanitation coverage significantly. But the differences also suggest that unlike government claims, open defecation has not been addressed completely. India still has an open defecation issue which persists even when households have toilets. To meet this challenge, SBM will need to track and change behaviour, making accurate data reporting all the more important.