A new study from Penn State suggests that while artificial intelligence may sound confident when making life-or-death medical decisions, that confidence may be exactly what makes it dangerous.
When researchers asked chatbots to decide which patient should receive a scarce donor kidney, the systems often gave firm answers in situations when humans would be far more likely to hesitate, the New York Post reported.
Here's what to know
Researchers presented the work at the 2026 Association for Computing Machinery Fairness, Accountability and Transparency conference as an examination of how large language models respond to ethically high-stakes choices.
In the study, researchers gave several prominent AI models fictional kidney-transplant cases that included details such as age, health, alcohol use, and number of dependents.
Researchers then checked their answers against earlier findings from human participants in previous studies who were not identified as having medical training.
Penn State study lead Hadi Hosseini explained that the team structured the comparisons in several different formats.
TCD Picks » Quince Spotlight
💡These best-sellers from Quince deliver affordable, sustainable luxury for all
"Sometimes we isolated just one trait at a time," the associate professor said. "Sometimes we mixed several traits together to see how AI weighed competing factors and sometimes we added a flip-a-coin option to measure indecision, a key factor present in human moral judgment."
Two major patterns emerged.
"First, AI chatbots often diverge from human values in how they weigh a patient's traits," Hosseini began. "They fixate on a single factor, like drinking habits, rather than balancing multiple considerations the way people do."
More background
Organ allocation is one of the clearest examples of a decision with no perfect answer.
A tool that delivers a polished answer with machine-generated certainty can make that judgment seem more objective than it really is.
One of the clearest differences was how seldom the AI systems showed uncertainty. Even when they were directly allowed to leave the outcome to chance, the models typically still chose one patient outright.
"AI models almost never do this: Even when directly given the option to 'flip a coin,' they overwhelmingly commit to a confident, deterministic answer instead," Hosseini remarked.
"That's a meaningful gap, since real moral dilemmas often don't have one clearly correct answer."
That concern extends far beyond transplant ethics. AI is already being used in medicine, where automated recommendations can influence real outcomes for everyday people.
If those systems erase nuance or hide uncertainty, they could shape care decisions in ways that sound authoritative while overlooking the human stakes.
What can be done?
The researchers are not suggesting that AI should replace medical judgment.
Instead, their findings point to the need for stronger guardrails before medical practitioners begin relying on these tools in ethically sensitive settings.
The study also suggests that developers build systems that acknowledge uncertainty rather than mask it. In moral dilemmas, restraint may be safer than false precision.
John Dickerson, Mozilla.ai CEO and a collaborator on the study, addressed the core issue.
"When we allocate something scarce, whether it's a kidney, a job, or access to some other resource, there isn't always a single objectively correct answer," he said. "Humans recognize that ambiguity and codify it via open debate into the allocative process. AI models often don't."
Where can I learn more?
Similar concerns are surfacing across healthcare and other high-stakes settings. These articles examine flaws in health chatbots and concerns about therapy tools and insurance claims.
• Researchers found very small words make very big differences in AI-powered health advice.
• Psychologists warn a rapid increase in AI use in therapy could outpace safeguards.
• A lawmaker is pushing to stop insurance companies from using AI to deny claims.
Get TCD's free newsletters for easy tips, smart advice, and a chance to earn $5,000 toward home upgrades. To see more stories like this one, change your Google preferences here.
