Between 1954 and 1973, a group of economics and finance professors completely changed the way the rest of us looked at the financial markets. They taught us some pretty basic stuff—you cannot earn higher returns without taking higher risks, financial markets are efficient, there is a reason why options trade at the prices they do, beating the market is not a mug’s game.

Time and Tested: Global investment firms and NYSE brokers have used Bernstein’s theory for best returns.

Their stories and their theories were told with insight and elegance by Peter Bernstein, a money manager and writer on finance, in his 1992 book, Capital Ideas: The Improbable Origins of Wall Street. It is one of the best books on modern finance.

Now, 15 years after that book was published, Bernstein returns to the same subject with Capital Ideas Evolving. Five of the economists covered in the original book have been awarded Nobel Prizes. It is now widely accepted in universities across the world that financial economics is worthy of serious study. And, perhaps most importantly, the ideas of people such as Harry Markowitz, Eugene Fama, Paul Samuelson, Fisher Black and William Sharpe have crossed over from theory to practice. Wall Street and financial parishes in other parts of the world now regularly intone words such as risk, return and volatility. It is all because of the work of these gurus.

Other theorists have joined them in the pantheon since 1992—Robert Shiller, for example, captured the headlines by denouncing the dot-com boom as a case of irrational exuberance, and that, too, when it was on rather than when the undertakers were carrying away the dead bodies of failed Internet companies. Bernstein has written on other recent economists who have polished and sharpened what is called the neoclassical theory of finance.

Capital Ideas: The Improbable Origins of Modern Wall Street by Peter L. Bernstein

Along with the spread of the original capital ideas, there has also been a recent and powerful challenge to its core assumption that investors are rational beasts. The neoclassical theory assumes that people are calculation machines —automatons that had unerringly judged risks and probabilities.

Two professors of psychology at the Hebrew University at Tel Aviv, Daniel Kahneman and Amos Tversky, have placed some dynamite sticks under the theoretical edifice built by the protagonists in Bernstein’s 1992 book. Kahneman, Tversky (and others such as Richard Thaler) have tried to show that there are limits to human rationality. We all prefer to use intellectual short cuts (or heuristics) rather than think about an issue or investment opportunity with complete rationality. These lead to investment mistakes and imperfect markets.

Bernstein takes it upon himself to defend his subjects against the criticisms of behavioural finance, though the defence is not always convincing. The debate between neoclassical finance and behavioural finance is still very new, and it is perhaps too early to say, for sure, which way the debate will be settled. Bernstein also does not pay adequate attention to the dramatic collapse of Long Term Capital Management (LTCM), a hedge fund that had two Nobel Prize-winning economists on its rolls and which almost pulled down the global economy in 1998.

LTCM was set up with the assumption of superior knowledge and the belief that powerful computer models churning out correlations between every conceivable type of asset class could lead to investing nirvana. The collapse of LTCM was not just about irrational markets and outliers, but also about the limits of financial knowledge.

Yet, there is little doubt that the economists whom Bernstein wrote about in 1992 have transformed the financial markets. Till these professors came along, picking winning stocks was a bit of a gentleman’s game—with a few crooks thrown in to add a bit of glamour to the show. How successful you were depended on whom you knew, what sort of inside information you had access to and how lucky you were. The old boys’ club has been replaced by geeks trained in financial theory, who man computers with powerful computational capabilities.

More than 80 pages of Capital Ideas Evolving are about the practitioners—investment firms such as Goldman Sachs Asset Management, Barclays Global Investors and Yale Endowment Fund, for example, that have used the insights of modern financial theory to deliver superior returns to their investors. That professional investors today worship alpha, or the extent to which a portfolio outperforms a benchmark index on a risk-adjusted basis, is a testimony to the spread of the core idea of neoclassical finance theory that all available information is already factored into a market price, thus making it difficult for an investor to beat the overall market without taking extra risk.

Capital Ideas Evolving is not quite in the same league as its predecessor, but that is partly because Bernstein is writing on current debates rather than those that have been settled years ago. But it’s still a very good book.

Capital Gains: From the stock market to the new age economy

These four classics demistify the ways in which markets function

Capital Ideas: The Improbable Origins of Modern Wall Street by Peter L. Bernstein

Still the best book on the academic research that helped find a method in the madness of modern financial markets. A must-read for those who want to understand the broad contours of modern financial theory without getting lost in the numbing mathematics that underlies it.

Fooled by Randomness by Nassim Nicholas Taleb

Written with the passion of a heretic, this book looks at the flawed way that most of us—both professional and amateur—go about investing. Taleb uses the insights of statistical theory, sceptical empiricism and behavioural finance to show how we mistake luck for skill.

Irrational Exuberance by Robert Shiller

This book came out at the peak of the Internet mania, when believers in the New Economy asked investors to suspend disbelief and participate in the digital revolution by buying online companies at stratospheric valuations. Shiller dismissed it all as a bubble in this book. No prizes for guessing who was right.

When Genius Failed: The Rise and Fall of Long Term Capital Management by Roger Lowenstein

Long Term Capital Management (LTCM) was a hedge fund set up with the assumption that superior mathematical models would yield superior returns. LTCM had two Nobel laureates on its rolls. A beautifully written tale about investing arrogance.