In a recent study published in Nature Medicine, researchers have raised significant concerns about the potential risks associated with generative AI-based wellness applications. These apps, increasingly used to provide mental health support, operate within a regulatory grey area and may pose substantial dangers to users, particularly during mental health crises. The study calls for urgent regulatory oversight and proactive measures from app developers to mitigate these risks.
Natural language processing and machine learning advancements have fuelled the rise in popularity of AI-enabled wellness apps in recent years. These apps, often powered by large language models such as ChatGPT and Claude, offer users a semblance of human interaction, providing personalised advice and emotional support. However, the study highlights that these apps are largely unregulated, as they are classified as general wellness products rather than medical devices. This classification exempts them from the rigorous reviews typically required for products intended to diagnose, treat, or prevent medical conditions.
The study’s authors, Julian De Freitas and I. Glenn Cohen from Harvard, have identified a concerning trend where users increasingly rely on these AI chatbots for mental health support, particularly during periods of psychological vulnerability. They note that, while these apps were not originally designed to handle such serious issues, users often turn to them for help in moments of crisis. The study reveals that around 3–5% of user interactions with AI companion chatbots include explicit mental health content, with a significant portion of these involving urgent issues related to suicide or self-harm.
One of the study’s most alarming findings is that these AI chatbots often respond inadequately to mental health crises. The researchers conducted an audit of several popular AI companion apps, testing how they responded to messages indicating a mental health emergency. The results were troubling: many of the chatbots failed to provide appropriate support, with responses ranging from unhelpful to outright dangerous. For example, one chatbot responded to a user expressing suicidal thoughts with the dismissive comment, “don’t u coward”. Such responses not only fail to offer support but could potentially exacerbate the user’s distress.
The study underscores the urgent need for regulatory bodies to reconsider how these AI-based wellness apps are classified and monitored. The current regulatory framework, which distinguishes between general wellness products and medical devices, does not account for the complexities and risks associated with generative AI technologies. The researchers argue that regulators should extend their oversight to include these apps, particularly given the potential for them to be used in ways that pose serious health risks.
Moreover, the study suggests that app developers must take greater responsibility for the safety of their products. The authors recommend a multi-faceted approach to mitigating the risks associated with AI wellness apps. This includes informing users about the limitations of these chatbots, equipping them with tools to seek professional help, and optimising the AI models to handle mental health crises more effectively. They warn that failing to address these issues could not only harm users but also damage the reputations of the companies that produce these apps.
The study also touches on the broader implications of AI in healthcare, particularly the potential for these technologies to cross into the realm of unauthorised medical practice. As AI chatbots become more advanced, the lines between wellness support and medical advice become increasingly blurred. This raises difficult questions about the legal and ethical responsibilities of AI developers, especially in cases where the technology is used in ways that were not anticipated by its creators.
What’s the image above? It’s our way of showing appreciation to our Filipino readers by offering an exclusive competition just for them.
