AI Mental Health Checks in Schools: Promise & Peril for Teens

The alert arrived around 7pm. Brittani Phillips, a middle school counselor in Putnam County, Florida, received a message from an artificial intelligence-enabled therapy platform used by students outside of school hours. The platform flags potential risks of self-harm or harm to others based on student text input. This particular alert was classified as “severe,” concerning an eighth-grader. Phillips spent the evening contacting the student’s mother, assessing the situation, and, notifying the police. This incident, from last spring, underscores a growing trend: schools are increasingly turning to AI-powered tools to monitor student mental health, raising questions about safety, privacy, and the role of technology in adolescent wellbeing.

Interlachen Jr-Sr High School, where Phillips works, utilizes Alongside, an automated student monitoring system, to address budget constraints and limited mental health staffing. The district has been using the platform for three years, and it’s part of a broader wave of at least nine companies receiving funding since 2022 for similar tools. Alongside claims to serve over 200 schools across the US, offering a combination of social-emotional skill-building chat – featuring a llama named Kiwi – and AI-generated content monitored by clinicians. Company representatives emphasize that the platform provides crucial resources, particularly for schools in rural areas, which often lack adequate mental health support. These resources are especially vital given potential cuts to mental health programs, particularly in rural areas.

The integration of AI into education is likewise a key component of the Trump administration’s national education agenda. However, this increased reliance on technology is meeting with resistance from some parents, educators, and lawmakers, who express concerns about increased screen time for teenagers. Several states have begun restricting the use of AI in telehealth, reflecting broader anxieties about its application in sensitive areas like mental healthcare. Concerns are growing about students forming emotional attachments to AI, with a recent national survey revealing that 20% of high schoolers have used AI romantically or know someone who has. This has even prompted a proposed federal law requiring AI companies to remind students that chatbots are not real people.

The Role of AI as a First Responder

Despite these concerns, Phillips believes the tool her school uses is effective at managing less critical situations. With approximately 360 middle school students to support, the AI platform helps address routine issues like breakups, allowing her to focus on students facing more serious crises. She also notes that some students find it easier to confide in an AI initially. “It’s exceptional at putting out the small fires,” she says.

School counselors suggest that student nervousness contributes to their comfort level with these technologies. Speaking with a mental health professional can be intimidating, particularly for adolescents, explains Sarah Caliboso-Soto, assistant director of clinical programs at the University of Southern California Suzanne Dworak-Peck School of Social Work. There’s also a generational aspect; students accustomed to chat interfaces through social media and websites may find AI interactions familiar. Linda Charmaraman, director of the Youth, Media & Wellbeing Research Lab at Wellesley Centers for Women, adds that students often find texting easier than phone calls and that chatbots eliminate the need to interpret facial expressions, potentially reducing feelings of judgment. Chatbots are available at any time, without the need for appointments.

“It’s almost more natural than interacting with another human being,” Caliboso-Soto observes.

Caliboso-Soto’s telehealth clinic has seen a rise in crisis text lines and chat lines, though it doesn’t currently use AI. She believes AI can serve as a valuable “first line of defense,” regularly checking in with students and directing them to further help when needed. Alongside’s services start around $10 per student annually, with discounts available for larger districts. However, she cautions against using AI as a substitute for human counselors, emphasizing the importance of human connection, judgment, and the ability to interpret nonverbal cues. “You can’t replace human connection, human judgment,” she adds.

Social Connection and the Limits of Technology

Sam Hiner, executive director of Young People’s Alliance, raises concerns about a broader societal issue: increasing loneliness and weakening communities. He argues that technology and social media have contributed to isolation, leading students to seek connection wherever they can find it, even through AI. Young People’s Alliance has released a framework for regulating AI that allows for some therapeutic uses, but generally prioritizes rebuilding human community and opposes AI use that replaces human companionship. Hiner warns of the dangers of “parasocial relationships,” where students develop one-sided emotional attachments to AI, particularly when used for therapeutic purposes. He stresses that AI should not convey emotions or imply sentience, as this can encourage attachment.

Hiner emphasizes that platforms often measure engagement without assessing whether it translates to genuine connection and wellbeing. He argues that the focus should be on fostering fulfilling, connected lives, not simply alleviating immediate feelings of loneliness. The key concern is preventing AI from eroding social skills and accountability, which are essential for healthy relationships.

Privacy concerns also loom large. Experts note that AI chatbots generally lack the same privacy protections as conversations with licensed therapists. Given heightened concerns about student privacy and potential police involvement, the use of these tools raises “messy” privacy issues, even with clinical oversight.

Phillips and Alongside representatives stress the importance of human oversight. Phillips believes the tool is an improvement over previous monitoring systems, which often led to disciplinary action rather than mental health support. This school year, she identified 19 “severe” alerts from the AI platform (out of 393 active users) as of February. She notes that some students test the system’s boundaries, intentionally typing concerning statements to gauge the response. These instances allow Phillips to assess the student’s intent and provide appropriate support.

The use of AI in schools is not without its challenges, but Phillips believes it can be a valuable tool when used responsibly and with appropriate human oversight. The potential for students to become overly reliant on AI is a concern that warrants careful consideration.

Looking Ahead: Regulation and Responsible Implementation

As AI continues to integrate into education, ongoing evaluation and refinement are crucial. The development of clear ethical guidelines and privacy protections is essential to ensure student wellbeing. Further research is needed to assess the long-term impact of AI-powered mental health tools on student development and social-emotional learning. The conversation surrounding AI in schools must involve educators, parents, students, and policymakers to create a framework that prioritizes both innovation and responsible implementation. The process will likely involve ongoing legislative review, refinement of platform safety protocols, and continued monitoring of student outcomes.