Home / Opinion / Columns /  How best to interpret CMIE’s consumer survey findings

As India’s official survey system battled censorship and apathy over the past few years, a private agency came to our rescue. The ‘consumer pyramids’ household survey of the Centre for Monitoring Indian Economy (CMIE) helped us peek into the life of an average Indian family. During the first lockdown, it was thanks to CMIE that we came to understand the scale of urban unemployment and distress migration of urban workers to rural farms. Official surveys confirmed these developments only much later.

The CMIE’s ability to publish data almost on a real-time basis and its freedom from state diktats have won it the admiration and approval of a growing tribe of economists and policy wonks. From evaluating poverty levels to estimating covid’s death toll, CMIE data has become the first port of call for many.

Yet, as the use of its database has grown, so has the number of questions about it. Economists Jean Dreze and Anmol Somanchi found that several demographic groups (the very young, uneducated and the asset-poor) find less representation in CMIE’s survey than in the latest National Family Health Survey (NFHS, 2019-21). Adding to these concerns, World Bank economists Suthirtha Sinha Roy and Roy van der Weide noted that even the very rich appear to find less representation in CMIE’s survey.

Economists Rosa Abraham and Anand Shrivastava found that CMIE reports fewer women in the workforce than the Periodic Labour Force Survey (PLFS). Women’s labour force participation rates estimated by CMIE are roughly half the ‘official rate’ estimated by the PLFS. CMIE shows a higher share of respondents with post-office savings, pension (or provident fund) plans and insurance products (life and health) compared to the All India Debt and Investment Survey (AIDIS) 2019, Niyati Agrawal and her colleagues at Dvara Research found.

Several economists and statisticians suspect that these discrepancies arise out of flaws in CMIE’s survey methodology and design. Writing for the India Forum, the statistician Salil Sanyal has raised pointed questions about CMIE’s stratification strategy (excessive sampling of urban areas), and its failure to adhere to statistical norms of probability sampling.

Sampling theory demands that households in the primary sampling unit (typically villages or urban wards) are chosen at random from a list of all households in that unit. A convenient alternative is to pick the first household at random, but use a fixed interval to select subsequent households. This sampling interval is calculated by dividing the total number of households by the desired sample size. So if a village has 300 households and a sample of 30 is to be drawn from it, the sampling interval is 10 (300/30). The surveyor would first pick a random number between 1 to 300. Let’s say that number turns out to be 25. Then the 25th household on the list will be the first to be sampled, followed by the 35th, 45th, and so on. Most of India’s official surveys use this technique as it saves time.

The problem with CMIE’s survey is that its field staff were unable to draw up a list of households in the primary sampling units because of practical difficulties (including security concerns in some locations) in the initial rounds. So CMIE used a ‘jugaad’ method to select households. The field staff were asked to enter the main street of a village, check the number of households on it, and pick a random number between 5 to 15 to select households. Once the main street is exhausted, enumerators move to the inner streets. The lack of a complete listing, the absence of a random start, and the use of an ‘ad-hoc’ interval (5-15) to select households inject biases in the sampling process, wrote Sanyal.

Other statisticians argue that CMIE’s household selection is problematic since residential arrangements in rural India are far from random. Richer households often tend to be clustered on the main street while poorer households could be clustered in a hamlet on the periphery of the main settlement. Economists Vikas Rawal and Jesim Pais have raised similar concerns. So have Dreze and Somanchi.

CMIE takes these concerns seriously and is currently investigating if its survey methods lead to any biases, said Mahesh Vyas, its managing director. In each village, the CMIE team is checking if the outskirts are being missed. The findings of this investigation will be published by October, and corrections to the sample (wherever necessary) will be effected from January to April 2023, Vyas said over email.

CMIE is also analysing discrepancies with official datasets, said Vyas, adding that this process will take some time. Critics who raise concerns about CMIE’s survey methods often use the NSS (National Sample Survey) as their frame of reference. But the CMIE survey is not conceptualized as a replica of the NSS, said Vyas. Still, CMIE is trying to build “some kind of a mapping" between NSS/NFHS definitions and systems and those used by CMIE, he added.

CMIE’s sincerity in addressing the concerns of data users is admirable. But given that these reviews and corrections will take time, it is worth treating CMIE’s survey numbers cautiously for now. Rather than rely on its survey numbers to measure levels of different economic indicators related to employment and living conditions, it may be prudent to use the data to track changes over time. Despite its imperfections, the survey is likely to capture big changes in unemployment or living conditions, as it did in 2020.

Pramit Bhattacharya is a Chennai-based journalist. His Twitter handle is pramit_b

Catch all the Business News, Market News, Breaking News Events and Latest News Updates on Live Mint. Download The Mint News App to get Daily Market Updates.
More Less
Subscribe to Mint Newsletters
* Enter a valid email
* Thank you for subscribing to our newsletter.

Recommended For You

Edit Profile
Get alerts on WhatsApp
Set Preferences My ReadsWatchlistFeedbackRedeem a Gift CardLogout