Making sense of Bihar’s forecasting fiasco
The latest round of poll bashing is a good coincidence to reiterate some basics about the state of polling in our democracy and what can be done to improve it
As I read Shekhar Gupta’s tweet, I realized that I was standing on the other side of a divide that I have known for long. In my previous calling, I was seen to be a “pollster” and was called upon to answer for polling disasters as and when they took place. Now I was an example of good old political judgment trumping over polls. This reversal of roles set me thinking.
Opinion and exit polls got the Bihar election horribly wrong. With the singular exception of the exit poll its sponsors were too scared to air, no one came close to suggesting a landslide for the Mahagathbandhan (MGB, or Grand Alliance). A majority of polls picked the wrong winner, the worst nightmare for any pollster. This category included the largest (sample size 76,000 + done across all 243 constituencies) and a very innovative exit poll (GPS-enabled tablets with an EVM (electronic voting machine)-like method to determine voting choices) commissioned by NDTV. The post-poll survey by Centre for the Study of Developing Societies (CSDS) was the closest to the real outcome, as it showed a four-point lead for MGB, but it did not make any seat forecast. (Full disclosure: I am on leave from CSDS, but have nothing to do with its surveys any more).
Political reporters did much better. Every serious journalist who came back from Bihar brought tales of how NDA had lost the plot. Rajdeep Sardesai and Ravish Kumar were willing to bet on a clear majority for the MGB. Surjit Bhalla translated this intuition into numbers and was the only one to speak about 175 seats for the combine. Based on their reporting and some common sense acquired over the years, I too ventured an informed guess. When Prannoy Roy was announcing the numbers for the NDTV exit poll, Shekhar Gupta looked unconvinced and looked to me for my reaction. I scribbled on a notepad “My gut sense: clear and comfortable majority for JDU+RJD+Cong”. Next day, 24 hours before the counting, I wrote on my Facebook page about why I disagreed with the polls and expected a clear victory for MGB. Shekhar Gupta graciously mentioned this on the counting day and tweeted about it.
This has led to the latest round of poll bashing, except that I find myself on the other side of the divide. Perhaps this is a good coincidence to reiterate some basics about the state of polling in our democracy and what can be done to improve it.
First of all, let’s begin by acknowledging that scientific opinion polling is a must in any modern mass democracy that goes through elections. Anecdotal field reporting and qualitative analysis are a must too. But if you wish to make sense of the public mood, these cannot substitute for systematic surveys with large samples. A casual conversation with a few dozen cannot substitute for a systematic conversation with a few thousand. Secondly, let’s remember that opinion polls are not just about election forecasting. The most valuable use of opinion polls is to understand public opinion in between two elections and to explain the how and why of voting. Let’s not judge the whole science of polling only by some seats’ forecast. Most politicians who condemn polls on television privately commission these very agencies to carry out polls for them. Thirdly, when it comes to election forecasting, India’s record is not very poor. We follow the First-Past-The-Post (FPTP) system where the relationship between share of votes and the number of seats is very crooked and difficult to estimate. Making exact seats forecast in such a system is very difficult wherever they follow FPTP. Allowing for this handicap, the record of some of the better polls is not bad. Polls, especially exit and post-polls, mostly pick the right winner and give a valuable sense of the public mood. They may not acknowledge it, but politicians and journalists have all learnt to adjust their ‘gut sense’ in tune with the findings of the polls.
Yet, Bihar is a timely reminder that all is not well. Before the current poll bashing leads to a mindless call for banning polls or something equally crazy, let me reiterate some of my old suggestions for improving the quality of polling and forecasting.
First, let’s learn to keep our expectations realistic. Except a fluke, there is no way anyone can forecast exact number of seats in an FPTP-based elections. If pollsters and anchors do not promise black magic, the viewers will also learn to scale down their expectations. They could begin by replacing words like ‘forecast’ with ‘estimate’ or ‘projection’. Second, let there be strong transparency norms about public disclosures for all polls. These disclosures could include not just sample size and sampling method, but also sample profile, method used to convert votes into seats and the sources of funding. Third, there should be a professional code of conduct and a court of appeal (ideally, set up by the industry or by the Election Commission) for violations of the code or scrutiny of rogue polls etc.
Finally, we need more and better research and development in the public opinion polling and election forecasting. It is silly to waste so much of national television time on something that is so under-researched. It is criminal to do so in a country that has produced world-class statisticians. All it would take is a five-year research grant, smaller than the major grants made by Department of Science and Technology, to a multi-disciplinary team. Let’s stop cribbing and, as they say, just do it.
The author is a founder-member of Swaraj Abhiyan, on leave from CSDS.
Editor's Picks »
- Hawaii street swallowed by ‘lava tide’ as many more homes burn
- Oil prices slump 3% as Opec and Russia consider output boost
- Uttarakhand Class 12, Class 10 results 2018 declared, check results at uaresults.nic.in
- Disney slashes forecast for ‘Solo’ as early ticket sales sag
- Delhi weather: Clear skies, maximum temperature likely to touch 45C
- Motherson Sumi continues to face margin pressure in foreign markets
- What the Warren Buffett indicator tells us about market valuations today
- Jet Airways lands with a thud in Q4 as fuel costs increase
- IBC amendments: Some dilutions, and a lot more speed
- Patanjali’s gambit is paying off in toothpaste wars