The imminent arrival of drones delivering parcels, of cars run by Google, and the looming prospect of robots taking over shop floors have led many to worry about the future of jobs. But such concerns are not entirely new. In one of his most famous essays titled, Economic Possibilities for our Grandchildren, John Maynard Keynes, arguably the greatest economist of the 20th century, first envisioned a future when machines will take over almost all the work humans do. Writing in 1930, at the height of the Great Depression, Keynes was sanguine about our society freeing itself from both want and work.

Keynes considered the huge improvements in productivity the industrial revolution had brought as a force for good, and argued that such forces would render much of human labour obsolete over time. While new technologies could displace labour faster than the pace at which society could find new uses for labour, Keynes argued that technological change would also improve living standards rapidly so that no one would have to live in want or misery.

“I draw the conclusion that, assuming no important wars and no important increase in population, the economic problem may be solved, or be at least within sight of solution, within a hundred years," wrote Keynes. “...for the first time since his creation man will be faced with his real, his permanent problem-how to use his freedom from pressing economic cares, how to occupy the leisure, which science and compound interest will have won for him, to live wisely and agreeably and well," wrote Keynes.

Keynes acknowledged that the disappearance of work from our lives may be painfully destabilizing, robbing us of a sense of purpose. But Keynes, who must have been in a really cheerful mood while writing that essay, argued that three hour shifts or a fifteen hour work week may yet put off the problem for quite some time.

The world changed rapidly in the years following the publication of Keynes’s essay. As industrialization created more jobs than it destroyed in the developed world and increased real wages across income classes, Keynes’s prediction was nearly forgotten. But growing wage inequality and the rising automation of jobs over the past few years have rekindled a debate on the future of jobs.

The idea that we may have to redefine our conception of work is gaining currency once again. An August report by the Pew Research Center based on a survey of technology experts pointed out that many experts believed that the changes in automation will “allow us to renegotiate the existing social compact around work and employment". The industrial age notion of what a job is could change, and it might mean less drudgery and more leisure for most people as more and more robots take up tasks humans are now doing.

To be sure, not everyone believes the robots are going to destroy more jobs than they create. Roughly half the respondents in the Pew survey believed that there will be more new jobs than before. But the very fact that there is a divide among experts today on the future of jobs is notable, and marks an important break from the past.

In the early 19th century, a group of English textile artisans calling themselves the Luddites staged a machine-trashing rebellion in protest against the mechanization in the textile industry because they feared they would lose their livelihoods. Since then, the term Luddite has been usually used in a disparaging sense, especially by economists, to describe those who stand against technological progress. The history of modernization and industrialization seemed to prove the Luddites wrong as rising productivity did not shrink the number of jobs; rather the workforce expanded in many countries even as more women started taking up jobs. The mechanization of farming took away many farm jobs in advanced economies but the industrial revolution ensured that those displaced found better paying jobs in factories.

But the past few years seem to have rekindled old fears about machines, and their impact on the labour market. Labour economists who had long forsaken the possibility that mechanization and automation could ever reduce aggregate employment are now beginning to have second thoughts.

There is a growing realization that robots and smart machines will obliterate some professions rapidly although experts still remain divided over whether their net impact over the long run will expand or shrink employment opportunities.

“Three possible scenarios could happen with this current wave of technology," said Andrew McAfee, associate director of the Center for Digital Business at the MIT Sloan School of Management in an interview published in Slate magazine. “One is that it is going to hit the economy, and it might take a while to work itself out, but in the end we will reach a happy equilibrium. The Industrial Revolution was great news, eventually, for British workers. Electrification of factories eventually led to a large, stable, and prosperous American middle class. That pattern should give us confidence that we will wind up in another happy equilibrium."

“Scenario two is that we see successive waves: artificial intelligence, automated driving that will impact people who drive for a living, robotics that will impact manufacturing. If scenario two happens, the problem is a bit worse because it will be difficult for the economy to keep adjusting and for workers to keep retraining."

“Scenario three is that we finally transition into this science-fiction economy, where you just don’t need a lot of labor."

“I believe that in my lifetime—I’m in my mid-40s—we’re going to see that third scenario," said McAfee. “We won’t see a zero-labor economy, but we’re going to head into a labor-light economy. Of course, people like me have been saying some version of that for 200 years. The Luddites, John Maynard Keynes, a lot of people have said it and been wrong. But when I look at the encroachment of digital stuff into the total bundle of skills and abilities that humans have, I think this time it is different."

Several economists share McAfee’s view that this time may be different and quite a few of them are less optimistic than McAfee about the future of jobs. In a 2012 National Bureau of Economic Research (NBER) working paper, economists Jeffrey D. Sachs and Laurence J. Kotlikoff argue that smarter machines will raise rewards for skilled workers disproportionately. Older workers, who have had the opportunity to pick up skills, will benefit by working in tandem with smart machines but the younger lot will suffer, the duo argue, and propose an inter-generational tax to compensate the young inexperienced lot. Skilled biased technical change will impact both skilled and unskilled workers but unskilled workers will be hit harder, the duo argues.

Sachs and Kotlikoff point out that the reluctance of economists to embrace the Luddite view springs not just from their analysis of the history of the Industrial Revolution, which benefited workers across the spectrum but also from the particular functional form economists use to model production in an economy. For the mathematically inclined, they refer to the Cobb-Douglas production function, in which different inputs enter the function symmetrically and hence technical change makes all inputs more productive. While this specific form may be very convenient mathematically and hence quite popular among economists, it is not necessarily the best description of reality.

In a widely cited study published a year ago, Carl Benedikt Frey and Michael A. Osborne of the Oxford University show that about 47% of occupations in the US face the risk of extinction because of automation. The risks are significantly higher for low-skill and low-wage jobs, according to their estimates. While routine tasks are at the greatest risk, even non-routine jobs requiring higher cognitive skills are not immune to the threat of extinction. Frey and Osborne argue that the advent of big data combined with lowered computing costs has meant that even tasks requiring cognitive skills can now be delegated to smart machines and machine learning algorithms which can compensate the lack of context and experience with brute processing power. The ability to analyze terabytes of data on past trends in a jiffy and their lack of bias can make smart computers more efficient than their human predecessors.

But are the doomsday writers overestimating the impact of automation and smart machines? Labour economist David Autor, who has spent long years researching the impact of technology on the labour market, certainly thinks so. In a recent research paper, the professor at the MIT economics department argues that the rise of smart machines will not just replace old jobs but also add many new ones, and will encourage greater specialization by human beings who will now be able to draw on the power of fast computing.

“...journalists and expert commentators overstate the extent of machine substitution for human labor and ignore the strong complementarities that increase productivity, raise earnings, and augment demand for skilled labor," writes Autor. “The challenges to substituting machines for workers in tasks requiring adaptability, common sense, and creativity remain immense."

Autor invokes the paradox put forward by the philosopher Michael Polanyi to underline his point that computers can never fully divine how human beings perform tasks because human beings themselves don’t explicitly know how they accomplish them. In Polanyi’s words, “we can know more than we can tell". Therefore, the rules for making decisions in many tasks cannot be pre-programmed, Autor writes.

“Tasks that have proved most vexing to automate are those that demand flexibility, judgment, and common sense—skills that we understand only tacitly—for example, developing a hypothesis or organizing a closet," writes Autor. “In these tasks, computers are often less sophisticated than preschool age children."

Autor describes two approaches used by computer scientists to deal with Polanyi’s paradox, or to computerize tasks for which we don’t know the rules. The first approach bows to the paradox, and accepts limits to which automation is possible. Autor cites the example of the Google car, which drives not on roads but on maps. If the car’s software detects that the environment in which it is operating differs from the pre-specified map (for instance, because of an unexpected blockade or detour), it will immediately ask the human driver to take charge. The second approach tries to circumvent Polanyi’s paradox by using the advances in techniques of statistical inference and machine learning to form best guess answers in cases where formal procedures are unknown.

“How well does machine learning work in practice?" asks Autor. “If you use Google Translate, operate a smart phone with voice commands, or follow Netflix’s movie suggestions, you can assess for yourself how successfully these technologies function. My general observation is that the tools are inconsistent: uncannily accurate at times; typically, only so-so; and occasionally, unfathomable."

Autor acknowledges that many of his examples are just prototypes which may get better with time. That indeed is the key selling point of machine learning evangelists, who argue that as computers mine more and more data, they get better. And one day, they will be able to make as good decisions as subject experts solely on the basis of empirical observations.

Indeed, the digitization of information and the gigantic explosion of workable data it has generated have led a growing tribe of analysts in fields as disparate as epidemiology and marketing to mine data for quick solutions to complicated problems. Machine learning algorithms may well replace these analysts over time. But the promise of machine learning and big data may be over-hyped. In many cases, establishing patterns and correlations are simply not enough; it is necessary to form theories or hypothesis, which can then be tested. It is often important to know the subject well even in order to mine data effectively.

Autor may, therefore, well be right in highlighting the complementarities that exist between smarter machines and a smarter workforce. But this also means that not-so-smart workers could still face the prospect of technological unemployment. If there is one thing that most experts agree on, it is on the rising importance of investments in human capital formation. Individuals and societies that invest more in education will continue to reap greater rewards even in the distant future.

Whether or not the future brings a shorter and more relaxed work week, as Keynes predicted, remains to be seen.​

Economics Express runs weekly, and features interesting reads from the world of economics and finance.