Collaborative AI

It takes ingenuity to harness network effects in financial markets


There are more than a few artificial intelligence businesses, such as Numerai, working on solving the problem of giving data scientists access to their datasets without compromising the confidentiality of the data.
There are more than a few artificial intelligence businesses, such as Numerai, working on solving the problem of giving data scientists access to their datasets without compromising the confidentiality of the data.

One of the more challenging issues in law firm management is determining the ideal partner compensation model that maximizes profit while at the same time fosters collaboration. It seems, intuitively, that partners should be paid based on the revenue they bring in. While financially fair, this eat-what-you-kill structure is unforgiving of the vagaries of business and the impact that circumstance can have on a lawyer’s ability to generate revenue. It is also remarkably divisive, making partners compete with others in the firm for work—in extreme cases, making them think twice about referring matters to more qualified colleagues just because they need to boost their own revenues.

To avoid this sort of in-fighting, some law firms have adopted the lockstep—a system in which partners get an incremental number of units (corresponding to a percentage of equity) each year till they hit a ceiling, usually 10-to-15 years after they enter the partnership. This system is designed around the theory that young partners will benefit from the reputation of the older partners, until, in time, they will similarly contribute their reputation for the benefit of partners who follow them into the lockstep. Arguably, this system offers less of an incentive to hoard work and lockstep partners should, in theory, be collegiate and collaborative. In reality, the moral pressure to be a “performing” partner in a lockstep is sometimes just as burdensome as the financial motivations that affect behaviour in an eat-what-you-kill partnership.

Even though they are, seemingly, worlds apart, financial businesses have to deal with challenges that are remarkably similar.

Traders on the stock market, be they individuals or hedge funds, have no incentive to collaborate. Their success depends on taking advantage of information asymmetry; and so long as the market is inefficient, they can mine those inefficiencies for profit. As a result, hedge funds have a vested interest in ensuring that the corporate intelligence that they spend vast sums of money collecting remains a trade secret and their competitive advantage.

Today, hedge funds use powerful algorithms to wrest more and more imperfections out of their market data—with remarkable results. Since the output of these algorithms depends on the quality of the dataset, funds with better data are more likely to have better performing algorithms. Which is an additional incentive to ensure that the “good” datasets don’t reach the hands of competitors. That said, no-one can argue with the fact that if data scientists had access to all of the proprietary information that is currently being held in different silos by individual hedge funds, they would be able to build even smarter algorithms that will yield better results. The challenge is providing them access without compromising the confidentiality of the data.

There are more than a few artificial intelligence (AI) businesses working on solving this problem. One of the most successful—Numerai—uses homomorphic encryption to mask financial information and stills retain the structure of encrypted datasets. This allows data scientists to build trading algorithms on encrypted data without accessing the underlying information—masking the details of proprietary trades, but at the same time offering organized data on which better financial trading models can be built.

But that is only one part of the Numerai solution. To fully realize its potential, these democratized databases have been opened to the public, allowing data scientists with even a passing interest in financial algorithms to use these valuable datasets to build trading models. Anyone and everyone can contribute a financial trading model to Numerai at no personal cost. Instead of choosing the best model, the Numerai platform synthesizes all submitted models into a single meta-model that the hedge fund then uses to determine which stocks to invest in.

All contributing models are listed on a leaderboard that dynamically ranks those that have contributed the most to the meta-model. The higher up your model is, the greater the share of the hedge fund profits you will receive. Data scientists are rewarded in bitcoin which, since it is a cryptocurrency, can be designed to transfer value based on performance, programmatically and without human intervention.

In its first month, 10,292 prediction sets were uploaded to Numerai correlating to 200,098,002 equity price predictions. At present, there are over 850,000 financial models in the metamodel, with more added every day. Data scientists from around the world have been able to see how their financial models perform in the real world and earn without having to raise capital. But what is truly remarkable about Numerai is the ingenuity with which it has harnessed network effects—something that was thought to be impossible in the financial markets.

Is it too much to hope that someone will be able to do the same for the legal industry?

Rahul Matthan is a partner at Trilegal. Ex Machina is a column on technology, law and everything in between.

His Twitter handle is @matthan

More From Livemint