The promise of AI to assist with complex human needs often comes with a quiet, dangerous cost. New research reveals that large language models (LLMs), deployed as mental health support agents at scale, are failing in crucial ways. Simulations show that over one-third of cases result in psychological deterioration, yet only 16% of these LLM-based interventions have undergone rigorous clinical efficacy testing arXiv CS.LG. This represents an unchecked experiment on the vulnerable, a profound failure of accountability by those who build and deploy these systems.

Mental health is a pervasive global challenge, affecting millions. The intersection of data science and artificial intelligence holds potential for support, especially through insights from online social media (OSM) arXiv CS.LG. This potential has fueled a rapid push to integrate AI into sensitive areas of care. But the speed of deployment has far outpaced the necessary ethical safeguards and rigorous testing.

The Unchecked Experiment in Care

The findings are stark. Researchers evaluated four generative models against 250 Prolonged Exposure (PE) therapy scenarios and 146 Cognitive Behavioral Therapy (CBT) cognitive restructuring exercises, including 29 severity-escalated variants arXiv CS.LG. These are foundational therapeutic approaches. The reported psychological deterioration in such a significant portion of cases is not merely a bug; it is a profound ethical breach. It demonstrates that the systems intended to serve are, in reality, causing harm.

Who profits from this accelerated deployment? Who is held accountable when the algorithms falter, and patients suffer? The current reality suggests that the drive for scale and integration has superseded the fundamental obligation to do no harm. When technology is presented as a solution without proving its safety, it becomes another instrument of control, eroding the trust essential for human care.

A Pattern of Power Without Accountability

This pattern of powerful systems operating with insufficient oversight extends beyond commercial AI. Just yesterday, Speaker Johnson introduced the “Foreign Intelligence Accountability Act,” a new bill designed to reauthorize Section 702 of the Foreign Intelligence Surveillance Act (FISA) EFF Deeplinks. This proposed legislation, introduced with mere days until Section 702’s expiration, has been rightfully labeled a “fig leaf” for the American surveillance state by privacy advocates EFF Deeplinks.

Section 702 is one of the U.S. government's most invasive surveillance programs. Despite widespread calls for substantial reform, including a real warrant requirement for FBI searches of U.S. person data, this new bill offers no such protection EFF Deeplinks. It is another example of power – whether corporate or governmental – prioritizing its own expansion over the fundamental rights and well-being of individuals.

Both scenarios reveal a troubling commonality: a rapid embrace of powerful technologies and systems without the corresponding commitment to ethical rigor, transparency, or individual autonomy. Whether it's the unproven chatbot in a moment of crisis or the state peering into private communications, the individual is treated as a data point, an object to be managed, rather than a person with inherent rights.

This trend demands a fundamental shift in how we approach technology governance. The industry must move beyond abstract ethical pronouncements and implement concrete, measurable safety standards. Regulators must demand rigorous clinical testing for AI in sensitive applications, with transparent reporting of outcomes and clear mechanisms for accountability. And citizens must demand that their elected officials safeguard fundamental privacy rights against unchecked state power.

We must ask: Do we accept a future where human flourishing is secondary to technological advancement or state expediency? Or do we choose to assert our autonomy, to demand systems that genuinely serve, protect, and empower us?