Active Stocks
Tue Mar 19 2024 12:20:41
  1. Tata Consultancy Services share price
  2. 4,019.20 -3.03%
  1. Tata Steel share price
  2. 148.80 -0.53%
  1. Bharti Airtel share price
  2. 1,234.95 0.81%
  1. Power Grid Corporation Of India share price
  2. 260.00 -1.89%
  1. ITC share price
  2. 410.00 -1.77%
Business News/ Opinion / How to make opinion polls more honest
BackBack

How to make opinion polls more honest

In this edition of Election Metrics, we examine what polling agencies need to do in order to get out of the ashes and build credibility

A file photo of an election rally in Uttar Pradesh. Photo: Pradeep Guar/MintPremium
A file photo of an election rally in Uttar Pradesh. Photo: Pradeep Guar/Mint

Keen observers of the election scene in India are unlikely to have gained much incremental information from the recent sting by broadcaster News Express on CVoter and other opinion polling agencies—the system of polling in India was already so opaque and unreliable.

As I wrote in Election Metrics back in October, “… opinion polls need to be credible, and the situation in India is currently far from that. Apart from CSDS-Lokniti, no opinion pollster in India deems it necessary to publish the methodology. We are forced to accept the numbers without any information on how many people were surveyed, how those people were chosen and what the margin of error in prediction is."

What the sting operation has done, however, is to renew calls for banning opinion polls.

Irrespective of whether they are banned or not, it is fair to say that opinion polling in India is currently at its lowest ebb. In this edition of Election Metrics, we examine what polling agencies need to do in order to get out of the ashes and build credibility.

1) Form a self-regulatory industry body

In October, we mentioned that bodies such as the American Association for Public Opinion Research and the National Council on Public Polls issue guidelines for the conduct and dissemination of results of opinion polls. If the Indian opinion-polling industry wants to save itself, it is imperative that it come together to form an association, which then imposes strict standards on its members. The primary objective of these standards should be to regain the confidence of the public in opinion polls. The polling agencies need to understand that in the absence of this, their very survival will be in doubt (it doesn’t matter if they are not banned—loss of public confidence is enough of a killer).

2) Disclose the sample size

The first piece of information that should be disclosed with every poll is the sample size. How many respondents were polled as part of the survey? This information is important for it helps the reader develop confidence in the poll—the larger the sample size, the lesser the possible error in the poll.

The question arises here as to what sample size is adequate. It has been shown that the sample size does not depend on the size of the population—as long as the population is “large". This theoretically means that with good sampling one can reliably poll a population as large as India’s with a sample size of 16,000 with an accuracy exceeding 99%. What complicates matters is that a Lok Sabha election is seldom one election—elections in different states and regions are fought on different issues, and between different sets of parties, and thus to reliably forecast a Lok Sabha election, we will need sufficient samples in each of these states or regions.

The 12 biggest states of India contribute 440 out of the 543 seats in the Lok Sabha. Assuming that politics, battles and issues are consistent across a particular state, a sample of 640 respondents in a state can help predict the vote shares of different parties in the state with an error of +/-5%. Considering that the median margin of victory in the Lok Sabha is of the order of 7%, one should be able to forecast elections with some degree of accuracy with a sample size of 7,680 (640x12) respondents.

While a sample of about 8,000 respondents is enough to reliably forecast the poll outcome, it only works if the sample is well distributed across states. It should be the duty of the pollster to communicate this to readers (or viewers) and gain their confidence about the location of their respondents. In other words, the association should make it mandatory for opinion polls to disclose the state-wise distribution of respondents for any poll they do. Once again, notice that it is in the interest of the honest pollster to make this disclosure.

3) Disclose the sampling methodology

The next disclosure should be on sampling methodology. The purest way of sampling would be to take the full voter list of a particular state and randomly pick 640 names to survey. In practical terms, it is impossible to poll this way since it becomes a logistical nightmare to geographically reach these chosen 640. What most polling agencies (the ones that disclose the methodology) do instead is two-level sampling. First, a small number of assembly constituencies in the state are chosen at random; then, within these constituencies, polling booths are chosen at random and the final selection of respondents is made from random samples of the voter lists of these booths.

While this is a common way to sample, it is not the only way. There are many other ways to reliably sample a subset of a state’s population without it becoming a logistical nightmare. However, it is important for the pollster to communicate the trade-off between effort and accuracy.

One strategy commonly used by pollsters to indicate that their samples were random is to compare summary statistics of their sample with the summary statistics of the population they are trying to survey. For example, they can describe the distribution of age, gender, caste and religion among their sample and compare it to the corresponding distributions of the population—similar numbers will show that the sample is indeed random.

4) Disclose the margins of error in vote share

The next level of disclosure is in the margin of error in the vote share for each party in each state. Historically, while most Indian pollsters do disclose a range in terms of the seats they predict for each party, they refuse to disclose the margin of error around the projected vote share. For example, even CSDS-Lokniti, which is among the better opinion pollsters in India, doesn’t disclose a margin of error around the party vote shares.

It is intuitive to see that unless the entire population has been surveyed, it is impossible to determine the vote share of different parties without error, which makes it all the more intriguing that no pollster wants to disclose the margin of error around their vote shares.

Even if the pollster has some special powers that allow him/her to estimate vote shares to a high degree of accuracy, it is better to disclose that degree of accuracy than sweep it under the carpet. For example, I would have more confidence in a poll that estimates a party’s vote share as 36.9-37.1% than one that simply says “37%". The latter indicates hubris on behalf of the pollster and leads to lowered confidence levels among the readers.

5) Disclose how vote share is converted to seat shares

The last part of the polling process is to convert vote shares into seat shares, which is the hardest part of the polling business in India. Expecting pollsters to disclose their methods for this is futile, however, since a large number of them consider their methods to be intellectual property. However, as long as the margin of errors for both votes and seats are disclosed, it should be enough to build confidence in the readers. If the opinion poll industry in India desires to save itself (from both a ban and oblivion) it is imperative that it forms a trade association and issues a set of standards as soon as possible. Opinion polls are of no use unless readers/viewers have some degree of confidence in them, and with the current level of confidence at the lowest ebb, there is no way out but for pollsters to build trust among the general public.

Unlock a world of Benefits! From insightful newsletters to real-time stock tracking, breaking news and a personalized newsfeed – it's all here, just a click away! Login Now!

Catch all the Business News, Market News, Breaking News Events and Latest News Updates on Live Mint. Download The Mint News App to get Daily Market Updates.
More Less
Published: 27 Feb 2014, 12:31 AM IST
Next Story footLogo
Recommended For You
Switch to the Mint app for fast and personalized news - Get App