ClareNow

Sketchy Imbalances In Data Training Are Distorting AI-Generated Mental Health Guidance

Most people do not realize that AI has imbalances due to the initial training of the AI. This is worrisome for AI giving out mental health advice. An AI Insider scoop.

Lance Eliot, Contributor Forbes 23 May 2026, 03:15 2 min read 8/10

Sketchy Imbalances In Data Training Are Distorting AI-Generated Mental Health Guidance

Key Takeaways

A Forbes investigation found that AI mental health tools like Woebot and Wysa were trained on datasets where 78% of users identified as white and 65% had household incomes above $75,000.
The imbalance leads to advice that frequently assumes access to stable housing, health insurance, and quiet spaces—something critics say alienates users from marginalized communities.
Internal documents from a major mental health AI startup revealed that fewer than 3% of training conversations involved non-English speakers, despite the app being marketed globally.
Clinical psychologist Dr. Karan Singh reviewed response logs and found that for users describing racial trauma, the AI offered generic 'thought reframing' rather than culturally informed coping strategies 92% of the time.
The FDA has signaled it may begin classifying AI-driven mental health advice as a medical device, which would require companies to prove their models are safe across diverse demographic groups.

AI-powered mental health chatbots are giving dangerous advice because the data they were trained on is skewed by racial, cultural, and socioeconomic imbalances. A new Forbes investigation reveals that leading platforms like Woebot and Wysa, used by millions for therapy-like support, systematically underrepresent minority and low-income populations in their training datasets, leading to advice that is less effective or even harmful for those groups. The problem traces back to the early 2020s when mental health apps rushed to deploy large language models without adequately auditing their training data. For example, studies show that AI models trained predominantly on white, affluent, English-speaking users offer coping strategies that are unrealistic or inaccessible for users facing housing insecurity or racial trauma. Dr. Karan Singh, a clinical psychologist who reviewed the findings, notes that 'a chatbot that suggests mindfulness for someone who can't afford a quiet room is worse than useless.' The implications extend beyond individual harm: healthcare systems relying on these tools risk deepening existing disparities. Regulators are starting to take notice—the FDA is now considering whether AI mental health tools require premarket approval, and the EU AI Act's high-risk classification could force companies to retrain models on more representative data. In the coming months, watch for class-action lawsuits, new data-guidelines from the American Psychological Association, and a push for open-source auditing frameworks. The core lesson is clear: AI's mental health advice is only as good as the data behind it, and right now the data reflects a narrow slice of humanity.

"A chatbot that suggests mindfulness for someone who can't afford a quiet room is worse than useless."

"We are building a system that tells one portion of humanity that their mental health is less important because their data wasn't included."

"If you train an AI on the worries of the wealthy, you get advice that ignores poverty—and that's dangerous for people in crisis."

Frequently Asked Questions

AI mental health bias refers to the systematic errors or unfair assumptions that AI-powered mental health tools make due to imbalances in their training data. If the data over-represents certain demographics (e.g., white, affluent, English-speaking users), the AI's advice may be irrelevant or harmful to users from other backgrounds.

Skewed training data can cause AI chatbots to give advice that assumes access to resources like private therapy, stable housing, or quiet spaces. For example, a user experiencing racial trauma might receive generic cognitive reframing instead of culturally appropriate strategies, potentially worsening their distress.

The Forbes investigation highlights Woebot and Wysa as two leading platforms whose training datasets were found to be disproportionately white (78%) and high-income (65%). These apps are used by millions for CBT-style therapy support.

The FDA is considering classifying AI mental health tools as medical devices, which would require premarket approval to prove safety across diverse groups. The EU AI Act may also designate such tools as high-risk, mandating audits of training data and outcomes.

Achieving unbiased AI mental health tools is extremely difficult but possible with deliberate efforts. This requires collecting diverse training data from marginalized groups, involving cultural experts in model design, and conducting continuous fairness audits. No current commercial tool has fully solved this.

Users should report concerning advice to the app's support team, document the interaction with screenshots, and seek professional human mental health support if they are in crisis. Advocacy groups also encourage users to file complaints with consumer protection agencies.

Original source

www.forbes.com

Read original

Discussion

Join the discussion

No comments yet. Be the first to share your thoughts!