The US Air Force has been unable to select a replacement for its ageing fleet of air refuelling tankers for almost a decade now. The selection process has been stalled several times by ethics scandals and legal challenges. Therefore, the authorities have decided to remove all subjective elements in the evaluation: 373 mandatory requirements have been laid out, each of which must be met on a simple pass-fail basis. In this effort to avoid any bias, the water flow in the aircraft’s toilets is rated as highly as the fuel offload rate.
Illustration: Jayachandran / Mint
One may laugh at this as an example of mindless bureaucracy. However, a similar approach to evaluation pervades the world of academics and economists too. A leading international business school has developed a Global Innovation Index (GII) to compare countries’ performances. Its model has 94 factors divided into eight categories. The mathematical means of the variables in each category are calculated to give the score of the category, and then a simple mean of the categories is calculated to determine a country’s rating. Mathematically very precise, but the 94 factors, though equally weighted, are very different. For example, they include “R&D expenditure as a % of GDP”, “Culture to innovate”, “Personal computers per 100 people” and “Political stability”. One may wonder if personal computers count as much as culture? Or political stability as much as research and development (R&D) expenditure? And how does one give a precise numerical score to “Culture to innovate” and “Political stability”? Similar intellectual weaknesses pervade international comparisons of “competitiveness” of countries. They list many factors which should matter and then conduct surveys of business executives’ opinions about them. The opinions are converted into mathematical scores and, voila, one has an accurate assessment of the country.
International comparisons of countries attract a lot of attention because we love competitions. We want to know who is winning. Therefore, we want a single number. However, policymakers should take such international comparisons with a huge dollop of salt. Most evaluations are an incomplete, and perhaps even inaccurate, representation of the phenomena that policymakers wish to influence, whether it’s innovativeness, competitiveness or even development of countries. The size and growth of gross domestic product (GDP) as determinants of how well countries are doing is a glaring example of the weaknesses in the single-number syndrome of measurement. While there is increasing dissatisfaction with GDP as an accurate measure of the overall health of a country, economists and policymakers continue to be almost obsessive about it. They have little respect for alternatives such as gross national happiness, which Bhutan and France are examining, because these indices include “subjective” matters that cannot be measured, they say. Nevertheless, the same policymakers seem to take other evaluations, such as those of competitiveness and innovativeness, quite seriously, even though these too are riddled with subjectivity.
Since all comparisons are suspect for one reason or another, one may as well as give credence to the 2009 Prosperity Index (PI) released by the Legatum Institute. This index is based on several criteria: not just economic wealth but quality of life issues as well. The structure of the index is very similar to the GII mentioned before. It comprises 79 different variables organized into nine sub-indices. The overall PI ratings are produced by averaging the scores of these sub-indices. Because India ranks much higher on social capital (trustworthiness in relationships and strong communities), personal freedom and (hold your breath) governance, too, the surprising conclusion is that India at the 45th rank is way ahead of China at the 75th position.
Policymakers need good models to produce good results. A model is a “system of things and relations” that “shows the construction of something”. Thus, the model of the innovation ecosystem must not only name and measure the various elements of the system, but it should also explain the relationships among them. So should models of competitiveness, development and prosperity. How do the elements within a model affect each other? What are the feedback loops among them? The size of the elements cannot be merely added up, as in effect they are when evaluations are made by adding the scores for each. And to leave elements out of a model merely because they cannot be quantified makes the model useless for predicting the behaviour of a system. This is the limitation of almost all economic models. The feelings and emotions of people are left out because they are “subjective”. However, when such mathematical models cannot explain the behaviour of a system, such as gyrations in the stock market, then analysts talk about subjective “moods” and “sentiments” as explanations.
The theme of the World Economic Forum in Davos this year was “Rebuild, Redesign, Rethink”. Financial, economic and environmental systems need rebuilding, redesigning and rethinking. We need to rethink how to think about systems. Human knowledge has advanced greatly in the last two centuries. And the advent of computers and the Internet in recent years has enabled the gathering and manipulation of masses of information. The problem is that knowledge is developing in silos, within ever narrowing specializations. Experts within these silos venture out too little. Thus, like the proverbial blind men around the elephant, they fail to see the animal. Linear deductive thinking has overpowered lateral systems thinking with grave consequences for our future as the threat of climate change reminds us. Interdisciplinary collaboration is required to develop good models that include all critical elements and also explain how these elements interact. We must rediscover the art of seeing the pattern and apprehending reality completely before we try to calculate our way through it. Because, ultimately, the quality of evaluation and planning depends on the quality of the models used.
Arun Maira is a member of the Planning Commission. Comment at firstname.lastname@example.org