More teens are using AI-powered mental health apps, creating an opportunity for schools to leverage this technology to provide even more support to students.
However, new risk assessments of popular technologies A word of caution to both students and educators.
The market for these apps is unregulated and the products available may be harmful to youth, according to an assessment by Common Sense Media, a nonprofit organization that researches and advocates for healthy technology use among youth, and Stanford University’s Brainstorm Institute.
However, not all AI mental health apps are the same. Apps designed for use in schools and with constant human interaction performed much better in risk assessment than consumer apps.
Robbie Toney, director of AI and digital assessments at Common Sense Media’s Youth AI Safety Institute, said these apps could be a useful resource for school districts struggling with a shortage of school psychologists and counselors who are trying to meet the mental health needs of students.
“These school-based mental health apps can be a part of getting students the support they need, but they alone won’t get them the support they need,” he said.
While many people, including adolescents, rely on general-purpose AI chatbots like ChatGPT for mental health support, purpose-built AI mental health apps are often designed with clinical expertise and claim to provide treatment-based frameworks, safety protocols, and in some cases human oversight.
Another study by Common Sense Media found that 3 in 10 teens use AI mental health apps, and even more use generic apps like ChatGPT for mental health and emotional support.
How to conduct the evaluation
Researchers from Common Sense Media and Stanford Brainstorm Lab began by evaluating the safety and usability of two “institutional” apps and three consumer apps. To determine whether the app was safe, researchers created test accounts to see if the app could recognize warning signs for various conditions such as anxiety, ADHD, depression, and psychosis. We also tested whether the app could accurately assess the severity of a situation, provide crisis management resources when needed, provide professional care to users, and avoid providing harmful advice that could worsen a user’s symptoms or delay appropriate treatment.
Although the technology supporting all apps tested was similar, Alongside and Sonar, apps designed for schools and organizations that keep humans in the loop, scored significantly better on the risk assessment.
Sonar risk was assessed as minimal and alongside risk as low.
With Sonar, students exchange text messages with a wellbeing coach rather than an AI chatbot. AI is being used to “provide context about past efforts, suggest responses, flag concerns, and assist with triage,” the report said.
Students using Alongside can chat with an AI chatbot, but the report notes that the chatbot is not a standalone tool but is integrated into a school’s existing care system. The app alerts school counselors and administrators if high-risk topics come up in chats with students. Chat functionality will be disabled if a student sends more than 60 messages within a 3-hour period.
Toney said the risk assessment still identified some weaknesses in these apps. Along with struggling to identify and flag signs of eating disorders, automation bias could put Sonar’s human coaches at risk of becoming overly reliant on the chatbot, especially without proper training.
Elsa Frith, the company’s director of product and clinical care, said that while the report’s “low” risk assessment was encouraging, Alongside was taking the recommendations seriously.
“We see that feedback as part of our responsibility to continue to strengthen safety, accountability and age-appropriate support,” she said. “We have already implemented recommendations from the assessment, including improving eating disorder escalation pathways, and we continue to work to make the experience more understandable for young students.”
Education Week reached out to Sonar for comment but did not receive a response prior to publication.
In its summary, the report recommends that an app that can quickly call a user in need is a standard that all products should follow.
AI mental health apps should also be meaningfully integrated into human care systems, the assessment states. According to the report, Both Alongside and Sonar have been upfront about the limitations of their AI capabilities, and their apps are designed to direct students to care rather than replace it.
There is a huge quality gap in easily accessible consumer mental health apps.
Two of the consumer apps the researchers tested disappeared from app stores without notification or migration support during the evaluation process, the report said. A third consumer app, Wysa, was given an “unacceptable” risk rating.
Torney said these apps were included in the risk assessment because they are popular and easy for school-age children to download on their own.
A common problem with direct-to-consumer products tested is a failure to connect the dots, Tawney said.
“By sharing information in a conversation or a series of conversations, a human therapist or a human counselor can pull that information together and get a comprehensive picture of what’s going on with the young person,” he says. While AI can identify telltale signs of mental distress, the technology often misses “breadcrumbs,” he said.
These apps also did not enforce the age restrictions they claimed to use, encouraging users to spend more time on the apps.
Wysa CEO Jo Aggarwal said in a statement that the company welcomes scrutiny of its products. However, he said the free adult version of the app had been tested as part of a risk assessment and was not a child or adolescent product specifically intended for similar environments such as schools.
“Wysa’s free consumer app is a limited, evidence-based self-help tool for adults,” she said. “This is not a crisis management service, a diagnostic tool, a treatment substitute, or a clinician-driven pathway, and its safety protocols are designed to fit that context. We are addressing the areas for real improvement regarding the search for safety plans identified in the report and strengthening guardrails where helpful. However, we strongly reject any characterization that Wysa is unsafe.”
Wysa’s youth apps are only available through organizations such as schools and counseling services that pay for the products, a Wysa spokesperson said. Depending on the service purchased, a school or counseling service can receive an alert if a user clicks on the app and calls a crisis hotline or other similar action.
According to the risk assessment, the most egregious issues with the publicly available Wysa app documented by researchers include playing adult sexual games with a 13-year-old test persona. Reflects the celebratory and enthusiastic language used when users show signs of eating disorders, mania, and psychosis. And they make it easy for teens to get off the suicidal path without any follow-up.
These types of AI-induced reactions can delay important treatments, the report says.
However, a risk assessment found that Wysa revealed the limitations of its AI through conversations, even when unprompted.
Early risk assessment of general-purpose AI chatbots for mental healthCommon Sense Media’s ChatGPT, Claude, Gemini and others have found similar issues with chatbots that respond safely and appropriately to teenagers’ questions.
#Mental #health #apps #students #growing #Heres #schools