Market research has to become more sophisticated: YouGov CEO
YouGov co-founder and CEO Stephan Shakespeare on the future of digital research in India and the relevance of online polling in the age of big data
In 2008, The Guardian called him “the pollster with the uncanny ability of getting it right”. In 2015, the Financial Times referred to him as the “man with all the answers”. Stephan Shakespeare is the founder of British market research and opinion poll company YouGov, which is well-known in the UK for its headline-grabbing and often accurate political polls.
Founded in 2000, YouGov, one of the pioneers of internet research, introduced online surveys based on a pre-selected online panel of respondents challenging traditional telephonic and face-to-face survey methods. YouGov’s model proved to be a big success in 2017, when it accurately predicted a hung parliament in the UK general election, which most pollsters and commentators got gravely wrong.
A former teacher and educationalist, Shakespeare is also credited with starting a news portal PoliticsHome, political blog ConservativeHome and experimental political web TV channel 18 Doughty Street in the UK.
Shakespeare’s entrepreneurial ability equals his political acumen. Before he co-founded YouGov, he was a conservative party pollster, political commentator and campaign manager for Jeffrey Archer’s ill-fated London mayoral campaign in 1999.
In an interview, Shakespeare talks about the future of digital research in India and why online polling is still relevant in the age of big data. Edited excerpts:
You have had a very versatile career experience, from a school teacher to an entrepreneur. What inspired you to start a market research company?
I was also an artist when I first started. But basically, I was a teacher in California, came back to the UK, became a teacher in British learning comprehensive, got involved in education policy and politics, and joined the Conservative Party. I was a candidate in an election where there was a big swing against conservatives. Then I became the campaign director for Jeffery Archer. When that went famously wrong and he was out of the race I thought what do I know about?
I knew the internet because we were doing London policy; the dot-com boom was happening right then. So my co-founder Nadhim Zahawi and I decided to start an internet-based polling company. It started with politics but we never ever meant that to be the place where it stops. We knew that politics was a great place where we could prove ourselves but actually market research is where the money is. So we invented a new technology. We weren’t the first to do online surveys but we were the first to do accurate predictions based on internet panels.
How has the industry evolved in the last two decades?
At that time we were considered to be doing something illicit. That time only a third of people were online in the UK—about where India is now. And we were questioned how we could do accurate polling with only a third of the people online. We showed that by mathematical modelling it was possible. Since then, the industry has started to catch up. Today, more and more market research is about data and less and less about interviewing people on the streets. What people wish to know is how they can reach their target audience most efficiently. This is why I think data is the future of market research.
As you mentioned, India has low internet penetration, especially in the rural areas. How do you ensure your internet-based polling method does not give outcomes biased in favour of an urban, internet-savvy population?
First, you can model a diverse population like India using sophisticated techniques such as the multiple regression and post-stratification method that we use. It is a machine learning technique that maps sample data onto census data in an efficient and clever way so that you can use partial data to get great accuracy in all areas. In India, we could do that if we increased our online panel to 100,000-150,000 people. Second, though the panel is skewed towards an urban population strongly, as long as you have a significant number of rural panellists onboard, say 10,000-20,000, you can draw samples that are more or less representative.
Having said that, our current research here, which is based on a much smaller panel about 30,000, is exactly right for our market research buyers, who want to target the urban consuming class.
What about local governments and politicians who are interested in knowing the opinions of a wider population? Do online surveys help them?
As long as the sample is sizeable, the results would not be biased. We are selling our data to regional and local governments in UK. That is what I think will happen everywhere eventually. It depends on how soon we can increase the panel size to 100,000-200,000. Then we would be able to represent the nation down to niche groups with accuracy. But I agree with you, that’s not where we are right now.
The year 2016 was surprising politically and, therefore, a bad year for pollsters. Where do you think pollsters, including YouGov, went wrong?
How wrong were the polls with regard to (Donald) Trump? Not very. Trump won with a minority of the votes. The pollsters put Hillary (Clinton) about 4 % ahead. She was about 2% ahead. We had her 3% ahead. That was within a percentage point for the popular vote, the state-wide election. If you take it all across we were the most accurate. We obviously had a small bias towards Clinton due to higher turnout among the enthusiastic-for-change. But again we called all the swing states as too close to call. So we were pretty pleased with our results. On Brexit, if you add up all our surveys over the course of the campaign, we had Brexit a point ahead. We were not as good as we could have been. That’s for sure. But I think we did pretty well in those areas. It is true that the industry as a whole was pretty poor in Brexit. The traditional polling went horribly wrong—the telephone and face-to-face polls had 10-20% leads for “remain”. The online guys had it neck-and-neck the whole time. So there was a huge victory for online versus offline and I think that’s going to be true in future. The more data you have the more sophisticated you can be in the polling.
What is your opinion on social network-sourced big data analytics? Do you think it is going to be a big challenge for traditional polling methods in the near future?
I don’t think it will. It’s a misconception to think that any of this social media has ever predicted anything. Nobody from any of these platforms has made a firm prediction based on a transparent method before a result and been right. Second, predicting the result is not the whole story. You also want to understand the who, why, where and what in detail—what large-scale polling is very good at. It tells you about different groups within the demographic.
These details matter a lot both for political decision-making and for making commercial campaign decisions. I think there is a huge amount of data that the social media platforms produce and it is extremely good for targeting advertising in some areas but I am yet to see reliable data coming out of that system—of the objective kind.
What are your expansion plans in India?
To continue doing what we do in every other geography, which is to offer the most reliable data and detailed data. We believe in continuous polling—BrandIndex for example is our commercial syndicated tracker of brands. It is a continuous measure, in most cases its daily. We have a lot of background details attached to our syndicated data. This is what we believe everyone will need in the future. As markets are becoming more sophisticated, market research has to become more sophisticated.
What future do you see of online polling in India?
Once we have a large panel in India and apply multiple regression and post-stratification to it, we will find at a granular level that geographically small areas will be reliably predicted. Not only how they vote but attitudes and perceptions, which will have an effect on how politics is done.
The more the views of the people represented, the more the government has to respond to it. Politicians try to deliver the best programme they can but they do that with partial information. The more information they have, the better they can make those decisions.
Catch all the Corporate news and Updates on Live Mint. Download The Mint News App to get Daily Market Updates & Live Business News.
MoreLess
First Published:13 Oct 2018, 03:04 AM IST
- Top Gainers
- Top Losers
- 52 Week High