No cure for bias in AI, we have to live with it: LinkedIn’s AI head Deepak Agarwal
LinkedIn’s AI head Deepak Agarwal on how the company harnesses AI, how it tackles AI bias, the role of the Bengaluru unit in this context and his views on fear of AI
Mumbai: Deepak Agarwal, head, artificial intelligence (AI), LinkedIn Corp., believes AI is the oxygen for all products and services on the world’s largest professional networking website. Agarwal leads a team of 350 people who continuously work on improving how users on Linkedin connect, find jobs and employees, and discover networking opportunities. In an interview, he spoke on how LinkedIn harnesses AI, how it tackles AI bias, the role of the Bengaluru unit in this context and his views on fear of AI. Edited excerpts:
How do you harness the power of AI at LinkedIn?
AI is a core component of every single product we build at LinkedIn. We call AI the oxygen of all our product features. When you visit the site, we help you connect with people, who is powered by the ‘people you may know’ feature, a purely machine-based recommendation that helps you connect with the right people. So, you can derive a lot of value out of the network. Once you have connected with people, you then need to nurture those connections, and that’s where Newsfeed recommendations become important. If you are looking at a product like jobs, it is again entirely powered by machine learning (subset of AI).
If you aspire to get a particular job but don’t have the skill set to do so, we recommend the best courses. This, too, is powered through recommendation and search on LinkedIn Learning. A large component of that, beyond content creation, it is powered by AI. If you are a recruiter and trying to source for a particular position, this recommendation is also powered by AI and machine learning.
If you were a sales person and trying to close a deal, the recommendations on decision makers you should reach out to, is powered by machine learning. If you are on the app, you might have seen a feature called Smart Replies. These are all examples of AI.
How do you go about the task?
Whenever we develop anything new on the app, the AI team sits with the designer team and the engineering team to design the new feature. We have been doing AI since 2006.
How challenging is this?
Machine learning can only be good if you have good data. So, at LinkedIn we are very fortunate enough to have a very rich data set. We have also built the LinkedIn Economic Graph based on data. There are 160 million variations of job titles that users use to describe their job descriptions and we have standardized all of that into 24,000. Similarly, there are so many different skills and the way people describe it. So how do you standardize and create a skills ontology and then classify every piece of content that we have on the platform to that skill ontology? This knowledge graph we have built out of unstructured data is raw material for a lot of work we do downstream: for recommendations, search, classification and so on. More recently, we have also been focusing a lot on making the end-to-end AI process as automated as possible. Every user has a personalised model, for instance, so given the scale of the problem making the end-to-end process automated is extremely important.
How complex is the whole process of filtering content?
Every AI model and every AI process is inherently biased. It is very difficult to create an end-to-end AI process which is completely unbiased. The only way you can mitigate bias is by collecting data as uniformly as possible. But as the dimensionality of the problem increases, you need an exponential number of more data points to solve the problem.
We have to live with bias in AI, there is no cure for it. What we should start doing is to eliminate harmful biases through the AI process. For example, if we are building a product for recruiting, we need to make sure that we are not inadvertently introducing gender bias. There is no silver bullet for this problem, but this is really our philosophy to solve this problem.
There’s a lot of talk about fake news, and privacy concerns by users and government. How does LinkedIn address this issue?
At LinkedIn, we are a member-first organization. When we build our product features or algorithms, we always think whether it is good for the members or not. If there are things we believe will hurt their privacy, we will not go ahead with it.
We sit down as a team and argue about it. We have a legal and privacy council and when we are in the process of building our product, their input is extremely important. Over time, the team has trained has us to think about these issues. This is more like a culture. It is more about making sure that it is infused into your culture.
How big is the AI team at LinkedIn and how many in Bangalore?
We currently have 350 machine learning scientists, researchers and engineers at LinkedIn working specifically on AI and machine learning and in Bangalore, we have 48. We have one centre each in Bangalore, Dublin, New York, San Francisco and Sunnyvale.
What about the computational power that is required?
You cannot build these sophisticated models without having enough adequate computational power. We have set up a GPU (graphics processing unit) cluster comprising around 150 GPU machines. We run roughly 200 AI online experiments every month. Roughly, we see about 1 in 10 experiments succeed. So 10x more experiments are prepared offline but 200 of them make their way to online every month. Our mission is to double that, keeping the same number of people. We want to every engineer at LinkedIn to be launching a successful experiment every month.
Is this why you have the AI academy?
We want AI and machine learning to be a part of every single product feature we develop. The supply and demand in this area is highly imbalanced. Hiring so many people with machine leaning expertise is becoming difficult. However, there is a lot of work in the process which does not need a Ph.D in machine learning. We thought it’s prudent for us to create a program that could train our engineers.
Our training is not focused on teaching them deep theoretical aspects of machine learning. For that, we recommend they take some courses outside and we support their effort. In the AI academy, we want to equip our software engineers more via a vocational training program where they can learn how to deploy end-to -end machine learning into production. The formal apprenticeship program is being better tested in the U.S and will be launching it soon in India.
Are we seeing integration of AI products and services with LinkedIn and Microsoft (LinkedIn is a Microsoft unit) teams sitting together and working on them?
We have been having a lot of interactions with Microsoft, both in AI and otherwise. If you look at MSR (Microsoft Research), they are perhaps the best research lab in the world. Microsoft is investing heavily into building the AI platform and we are leveraging a lot of those. Recently in Bangalore, we have been able to integrate their multimedia technology, which we are now using to analyze video, speech to text, image analysis, etc. We didn’t have to build these things from the scratch--all of this was available through Microsoft and then we are tailoring and customizing it to our use.
We have been using Microsoft’s Machine Translation APIs, for instance, and you will find a lot of that goodness on the platform itself. We have benefitted a lot from the investment. There was a product that we launched called the ‘Resume Assistant’. As you are writing your resume in Microsoft Word, we will help you write a better resume, using all the LinkedIn data. Similar there have been integrations with Outlook and LinkedIn.
Tell us a little more about the goal of the Bengaluru centre?
Bengaluru is a very important centre for us in terms of doing AI development work. We started this team three and a quarter years ago and today we have 48 people in three years, which is really impressive. The most important area that Bangalore is focusing for us here is how do we take all the multimedia content and extract signals out of that which can then be used across the board by all products (newsfeed, advertising, learning, search, etc.). In the process they are building technology and leveraging whatever we could from Microsoft. Therefore, it is a pretty interesting and challenging area and is entirely run out of Bengaluru.
The other big focus area is to keep our application and ecosystem safe and clean. There is a lot of unprofessional content shared on the newsfeed every single day. The more LinkedIn has grown in popularity and usage, there has been more vectors of attack on the platform. I think that is one big focus for the Bangalore team, and they have been doing this for 2 years now. That was the first project since they started out.
How many women do you have in your AI team?
We did an analysis late last year, which indicated that almost 17-18% of our team is composed of women engineers. We are trying to hire more, and trying to increase the diversity in our team.
Do you think companies should have a separate AI executive who reports to the board, given the importance of AI in the whole digital transformation of companies?
My personal opinion here is that it depends on your product. If your company is building a product that is very AI-centric and very data-centric, AI should play a very prominent role. And whether the person reports to the board or CEO, they should obviously make an arrangement where this role is very visible and involved in the day-to-day decision making of the product. But if you are building a product that is not very AI-centric, then obviously the arrangement could be very different.
I think in five years, I can hardly imagine a technology area where that won’t be the case. So, every company should start thinking about this, even if they are not using data today to improve their product. For instance, if you are into manufacturing, you may think that AI does not play a big role. But in the future, I can imagine robotics and AI getting together. There will be a software piece that will be collecting data and there will be a robotics piece, and they can all combine together and we can make the entire manufacturing process more efficient and more high quality by using AI and data.
So, even companies like that who have not been traditionally using AI and data in a big way, need to start thinking more strategically, and carve out that role
What about the fear of losing jobs to AI bots and robots?
At the end of the day I believe, just like any other powerful technology, AI is also one. How we use it is going to determine if it’s going to help us or hurt us as a society. Yes, it is possible that through AI and automation, some of the traditional jobs may not be needed. Those people, then, need to upskill themselves and do jobs that require different skill sets and different cognitive abilities. However, the same AI is helping us do that. If you look at the LinkedIn platform, we are helping you upskill yourself, helping you bridge that information asymmetry that exists.
There are a lot of jobs out there in the world, and there is a lot of talent out there. But today we are not able to match those at scale because of information asymmetry. And this is exactly what platforms like LinkedIn do. Again, this is all powered through AI. So, the same AI which might actually reduce some jobs, actually opens up new opportunities.
Do you believe AI will become sentient any time soon?
Given where the technology is today, I definitely don’t subscribe to that point of view. There are still very simple things today which are not possible only through AI. Of course, AI is a very powerful technology. It helps you gain a lot of insights from your data and helps you solve problems where you have clearly defined what your objective is. But getting to a point where it learns itself and figures out things to do on your behalf, without you even being able to control it-I don’t know. I can always make a prediction given that I am not held accountable for it and say whatever I like. Realistically speaking, I think we are not even close to it.