Companies Want Workers to Trust AI—But Not Completely

Companies say they are moving forward with AI use cases, especially in areas like code generation or summarizing documents or audio recordings, even as they acknowledge the technology’s potential for inaccurate results, bias, and so-called hallucinations.

Companies say they are addressing those challenges by ensuring that human workers check over everything the AI generates, but they are also concerned that those humans could start letting things slide or perhaps not fully understand the role they are required to play.

“If you put an AI in there, and it works five times in a row, I can see it’s like: if you don’t double check it, what’s the worst that can happen?" said Rosalia Tungaraza, Baptist Health’s assistant vice president of artificial intelligence.

Now companies are formulating new protocols and urging AI tool providers to factor an opportunity for double-checking into the workflow. Their goal is to avoid the potentially litigious stakes of letting unchecked AI-generated content run free in company documents, code or customer-facing material.

At Baptist Health, Tungaraza said that in instances where AI is summarizing recorded conversations between doctors and patients, she trusts the doctors to diligently double check since their career and reputation are on the line. In other areas of the business, such as operations or call centers, where AI could be used to summarize customer-service calls, there is more of a risk that it won’t be diligently double checked, she added.

Despite an overwhelming demand for AI skill sets and deployment, only 13% of employees have been offered any AI training in the past year, according to a survey from staffing agency Randstad.

Tungaraza said that for some use cases, the health system might consider enforcing double-checking, in the form of a signoff, as well as having consequences where the signoff isn’t completed.

At Workday, a provider of enterprise cloud applications for finance and human resources, Chief Technology Officer Jim Stratton said he has heard from a number of customers that generative AI tools need to make themselves obvious to users and offer the opportunity for double-checking.

Part of Workday’s priorities as it develops AI tools include making sure that it is obvious to the end user that they are working with generated output. “They need to review the results and go through it and make sure it’s proper and safe and correct, before pushing it out into the wild," Stratton said.

While the trust in AI models is increasing, final approval should rest with human employees, said Lea Sonderegger, chief digital and information officer at crystal and jewelry maker Swarovski. Checking over output must become part of the framework in how employees use AI, she added.

Currently, there is no official legal governance to enforce how and where companies need to have employees to double check AI output, and each company is working through its own systems and workflow—although ensuring some kind of checking will be important, said Ricardo Madan, senior vice president at TEKsystems Global Services, an IT services and consulting company.

“If you don’t have humans fact checking, governing and putting this into some sense of reality, then you’re missing it," Madan said. “We’re creating that process right now jointly with customers."

As the technology gets better and more accurate, the critical need to have everything double checked will also decrease, said Juan Perez, chief information officer at Salesforce.

Perez said some cases, such as those in the medical field, may always require checking, no matter how good the model gets.

He added: “But, I think that there will be some use cases in the future where the models will be trusted enough, smart enough, and capable enough where the need for the human to do the checking may not necessarily be there."

Write to Isabelle Bousquette at isabelle.bousquette@wsj.com