Data reliability: Lessons from the 75-year-old National Sample Survey

“It is not often recognized that behind every piece of statistical information lies the dedicated work of hundreds of NSS investigators who have conducted painstaking surveys and interviews, often in remote areas," he said.

The 1999-2000 NSS consumption expenditure survey suggested that poverty had declined sharply compared to the mid-90s, Vajpayee pointed out. Had any politician made this claim, it would have been questioned, he said. But nobody could question the authenticity of the NSS survey, he added.

“Governments come and go, but an autonomous organization like yours functions without being affected by political and governmental changes," said Vajpayee.

Also read: Statistical reform has begun: How far will it go?

While Vajpayee was right about the value of NSS data, he picked the wrong example to highlight his point. The 1999-2000 consumption data turned out to be extremely contentious. NSS had faced criticism in the 1990s for failing to capture the country’s changing consumption trends.

In response, the NSS team ran a series of trials to modify its questionnaires. Even before the experiments could reach their logical end, the 1999-2000 round questionnaires were revised. The hurried revisions led to a flawed survey.

Eventually, that round was deemed incomparable with other NSS rounds and later excluded from the Planning Commission’s official poverty estimates.

The 1999-2000 fiasco was a rare exception in an otherwise stellar run of NSS surveys since the 1950s. All governments in India, irrespective of ideology, have relied on NSS data for framing development schemes and assessing their impact.

The early work of NSS pioneers helped shape statistical practices across the globe. As we celebrate the platinum jubilee of the NSS, it is worth reflecting on three key lessons from its unique history.

The first lesson pertains to the importance of public-spirited innovation: The NSS took shape as a collaborative venture between the Government of India and two academic institutions: the Indian Statistical Institute (ISI), Kolkata, and the Gokhale Institute of Politics and Economics (GIPE), Pune.

Its early rounds not only inspired similar experiments in other parts of the developing world, but also led to a series of experiments at the US census bureau in the 1950s, leading to substantive changes in the 1960 census.

The global impact of the NSS owes a lot to the pioneering band of researchers led by P.C. Mahalanobis, who crafted it. But India’s political leadership also deserves some credit.

India’s first PM Jawaharlal Nehru and finance minister C.D. Deshmukh backed a team of ‘lateral entrants’ led by Mahalanobis in their efforts to build a world-class data gathering apparatus. The backing of the Indian state and the NSS team’s commitment to excellence both played a part in making it a role model for other countries.

Also read: Data deficiency: India needs to map its informal economy better

The second lesson pertains to autonomy: The initial rounds of the NSS were supervised by ISI researchers. Once the National Sample Survey Organization (NSSO) was set up as a government body in 1970, an autonomous governing council began supervising it.

In the early years, the council was led by economists of unimpeachable integrity (such as V.M. Dandekar and B.S. Minhas). Data users were able to trust NSS reports because they knew that survey decisions were not shaped by a partisan agenda.

The 1999-2000 round marked a rare occasion when the governing council failed to insulate NSSO operations from the politics of the day.

In the words of Nobel-winning economist Angus Deaton, the fierce debate between the economic left and the economic right on the impact of economic reforms “intensified the debate on survey design and led to an unfortunate compromise design that temporarily undermined the poverty monitoring system".

The third lesson pertains to the limits of a centralized model of data collection: While the success of NSS spawned similar experiments globally, no Indian state was able to build a survey organization of similar standing. State statistical bureaus did take part in NSS rounds from the 1950s itself.

But the data they collected was rarely used for policymaking. It was the ‘central sample’ that formed the basis of all NSS reports. The ‘state sample’ data was largely viewed as untrustworthy, even by state-level policymakers.

Mahalanobis’ original idea that the state data would help cross-check the central data never became a reality.

Once the early pioneers left the scene, the spirit of innovation waned.

In 2001, the Rangarajan commission report observed that “the scientific approach of problem solving through analytical studies and pilot experiments, for which the NSS was well known in its early days, has been given up under the pressure of day-to-day work."

Also read: P.C. Mohanan: Tapping data to define the India story

Had the NSSO developed a collaborative model of data collection with a network of universities and research centres spread across the country, it might have created a more diverse and innovative data ecosystem.

Such a system would have generated better feedback loops for improving survey methods and designs. State-level policymakers might have been able to find credible local data-sets more easily.