In 2016, a Mercedes-Benz executive said that in developing autonomous vehicles (AVs), the company would prioritize the safety of its customers over that of bystanders and other drivers: “If you know you can save at least one person, at least save that one. Save the one in the car.” The company probably did not expect the media backlash that ensued. “Mercedes-Benz admits automated driverless cars would run over a CHILD rather than swerve and risk injuring the passengers inside,” a Daily Mail headline announced. Within a week, the carmaker publicly backed away from its position, stating that “neither programmers nor automated systems are entitled to weigh the value of human lives.”
Of course, human drivers already make implicit trade-offs when it comes to safety. We often prioritize not just our survival but our convenience, as when we fail to stop for a pedestrian who is clearly intending to enter a crosswalk or when we make a rolling stop to save time at a stop sign. Yet many people are clearly uncomfortable with the idea of AVs that explicitly encode similar preferences. For carmakers developing self-driving vehicles, the important question is: Would customers actually refuse to ride in “selfish” AVs?
In new research, I asked 5,584 participants to consider a situation in which an AV facing a close call must decide whether to prioritize saving its own passenger or a pedestrian. People were more outraged, on average, by the idea of an AV preferring to save its passenger, even when asked to imagine that they themselves were the passenger. When participants were asked to decide how an AV should be programmed to handle such trade-offs, 27% said it should save the passenger and 19% the pedestrian, but 54% said the vehicles should have no preference.
But would consumers truly prefer to buy “egalitarian” AVs, or are they simply afraid of sounding selfish if they say otherwise? In a follow-up study, participants were asked to imagine a scenario in which an AV must choose whether to strike a pedestrian or swerve and run into a barrier. They were then asked whether the AV should choose at random or follow a rule that it will strike whichever is closest, the person or the barrier.
When I described a scenario in which following that rule would mean hitting the barrier, thus endangering the passenger in the car, only 47% of respondents said the car should follow the rule. But when following the rule would lead to hitting the pedestrian and sparing the passenger, 62% of respondents endorsed it. It appears that people are inclined to choose AVs that will prioritize themselves over a random bystander, as long as they are given an unselfish alibi—a way of explaining their choice that doesn’t make them look bad.
These findings suggest that carmakers would benefit by avoiding overt appeals to customers’ self-interest when discussing safety design. For instance, their marketing can employ language that insinuates self-interest—for instance, “Putting you first, no matter what”—rather than exposing it directly—“Putting you before others.” Indeed, in a study offering these specific slogans, more than twice as many participants said they would ride in an AV marketed with the vaguer slogan than the overt one.
In January 2022, Tesla followed this principle when it began allowing customers to choose whether their car’s driver-assistance system drove in “chill,” “average” or “assertive” mode. In assertive mode, the vehicle maintained a smaller following distance, remained in the overtaking lane and performed rolling stops. Even if these choices are selfish, the term “assertive” allows Tesla owners to offer plausible alibis, telling themselves “my vehicle should drive confidently” or “driving assertively is safer.”
Even so, in February 2022 Tesla was forced by the National Highway Traffic Safety Administration (NHTSA) to recall 53,822 of its vehicles, after the rolling-stop behavior was deemed dangerous. A year later NHTSA doubled down, requiring Tesla to recall 362,800 vehicles because its driving software could “potentially infringe upon local traffic laws or customs.”
The experiences of Tesla and Mercedes suggest that as AVs become more popular, companies that appeal to consumers’ selfish preferences risk jeopardizing the full safety promise of this technology.
In a final study, I gave participants a different alternative. What if, instead of explicitly choosing between a passenger and a pedestrian, the AV is programmed to minimize the risk of any death occurring? In this scenario, participants overwhelmingly preferred minimizing overall risk to choosing which life to spare. This suggests that there are ways of engineering and marketing AVs that need not appeal to consumer’s selfish impulses.
Julian De Freitas is director of the Ethical Intelligence Lab at Harvard Business School.