Picture Sarah, a name I've chosen to represent countless individuals, waiting. Her life hinges on a decision made not by a compassionate committee, but by an algorithm. An algorithm designed to allocate a life-saving organ. When that system makes a choice starkly different from what human doctors, human families, or Sarah herself would deem just, who is accountable? And what have we truly gained?
For too long, we have been told that technology will enhance human capabilities. Yet, a stark, uncomfortable truth is emerging: Artificial intelligence, even when developed with good intentions, is quietly undermining human moral values and amplifying systemic biases in critical decision-making. Two new pre-print papers, published recently in arXiv, lay bare these critical tensions. One reveals how Large Language Models (LLMs) diverge from human moral choices in high-stakes scenarios arXiv CS.LG, while the other exposes how efforts to secure data privacy can inadvertently introduce new forms of unfairness arXiv CS.LG.
The Machine's Moral Calculus
Take the scenario of organ allocation. This isn't a hypothetical future; it is a present concern, demanding our immediate attention. Researchers systematically evaluated prominent LLMs against human preferences in kidney allocation scenarios arXiv CS.LG. The findings raise profound questions about the transfer of such critical decisions to algorithmic systems.
The LLMs exhibited “stark deviations from human values in prioritizing various attributes,” according to the paper arXiv CS.LG. This means that in situations where human lives hang in the balance, these supposedly advanced systems would make choices fundamentally different from what the people they serve would want. We must ask: who decides which human values are programmed into these systems? When LLMs are integrated into such critical processes, they are not merely assisting; they are often dictating outcomes. The profound ethical questions about accountability and human oversight become undeniable.
Privacy's Unintended Toll on Fairness
While some AI systems actively misalign with human values, others, intended to protect, inadvertently create new harms. Differentially private learning is a crucial technique for training models on sensitive data, aiming to protect individual privacy within large datasets. Yet, new research reveals a significant trade-off. Differentially private stochastic gradient descent (DP-SGD) “can degrade performance, introduce fairness issues like disparate impact, and reduce adversarial robustness” arXiv CS.LG.
This means that a technology designed to safeguard privacy, often hailed as a solution to data exploitation, can simultaneously create and amplify bias. It introduces unfair outcomes, disproportionately affecting certain groups. This is not a bug that can be patched with a quick fix; it is a fundamental tension embedded within the system itself, a structural problem where one form of protection comes at the direct expense of another. When companies deploy these systems, they must acknowledge that the pursuit of privacy, however noble, cannot excuse the quiet proliferation of disparate impact.
Industry's Choice and Our Responsibility
These findings challenge the easy optimism that often surrounds AI development. They expose a deep debt of accountability incurred when powerful technologies are integrated without rigorous, independent scrutiny of their real-world impact. It is no longer sufficient for developers and companies to claim they are building “private” or “intelligent” systems. They must prove these systems are also just, equitable, and aligned with fundamental human values.
For too long, the phrase “it’s complicated” has served as a shield for inaction. But these papers reveal that the complexity is often manufactured, or at least, predictable. The consequences are real, impacting individuals' autonomy, health, and fair treatment. Companies must prioritize transparency, invest in comprehensive ethical audits before deployment, and empower independent researchers to scrutinize their models' impacts. Regulators must demand clear accountability for algorithmic harms, especially when human lives or fundamental rights are at stake.
The choice, as always, remains ours: will we allow technology to decide for us, or will we demand that it serves human flourishing? The ability to choose – to say no, to demand better – is what separates a person from a product. If we allow algorithms to quietly reshape our moral landscape, what then defines our humanity? What then separates us from the machines we create, if not our insistence on choice?