I recently posted on Twitter the strong recommendation that individuals do not attempt to use large language models (like BERT or GPT-3) to replace the services of mental health professionals. I’m writing this blog post to specifically address individuals who think that this is a good idea.

CW: discussions of mental illness, medical abuse, self-harm and suicide. If you are in the US and currently experiencing crisis, 988 is now the nationwide number for mental health, substance use and suicidal crises, if you are outside the US this is a list of international suicide hotlines.

First: I want to start by acknowledging that if you are in the position where you are considering doing this, you are probably coming from a well-meaning place. You may know someone who has experienced mental illness or experienced it yourself and you want to help. That is a wonderful impulse and I applaud it. However, attempting to use large language models to replace the services of mental health professionals is *not* better than nothing. In fact, it is worse than nothing and has an extremely high probability of causing harm to the very people that you are trying to help. And this isn’t just my stance. Replacing clinicians with automated systems goes DIRECTLY against the recommendations of clinicians working in clinical applications of ML in mental health:

“ML and NLP should not lead to disempowerment of psychiatrists or replace the clinician-patient pair [emphasis mine].” – Le Glaz A, Haralambous Y, Kim-Dufor DH, Lenca P, Billot R, Ryan TC, Marsh J, DeVylder J, Walter M, Berrouiguet S, Lemey C. Machine Learning and Natural Language Processing in Mental Health: Systematic Review. J Med Internet Res. 2021 May 4;23(5):e15708. doi: 10.2196/15708. PMID: 33944788; PMCID: PMC8132982. 

Let me discuss some of the possible harms in more detail, however, since I have found that often when someone is particularly enthused of using large language models in a highly sensitive application they have not considered the scale of the potential harms or the larger societal impact of that application.

First, large language models are fundamentally unfit for purpose. They are trained on a general selection of language data much of which has been scraped from the internet, and I think most people would agree that “random text from the internet” is a very poor source of mental health advice. If you were to attempt to fine-tune a model for clinical practice, you would need to use clinical notes. These are extremely sensitive personal identifying information. Previous sharing of similar information, specifically text from the Crisis Text Line for training machine learning models, has been resoundingly decried as unethical by both the machine learning and clinical communities. Further, we know that PII included as few as one time in the training data of a large language model can be re-identified via model probing. As a result, any of the extremely sensitive data used to tune the model has the potential of being leaked to end users of the model.

You would also open patients up to other kinds of adversarial attacks. In particular the use of universal triggers by a man-in-the-middle attacker could be used to intentionally serve text to patients that encouraged self harm or suicide. And given the unpredictable nature of the text output from large language models in the first place, it would be impossible to ensure that the model didn’t just do that on its own, regardless of how much prompt engineering was done.

Further, mental health support is not generic. Advice that might be helpful to someone in one situation may be actively harmful to another. For example, someone worried about intrusive thoughts may find it very comforting to be told “you are not alone in your experience”. However, telling someone suffering from paranoia who says they are being followed “you are not alone in your experience” may help to reinforce that belief and further harm their mental health. Encouraging someone suffering from depression to “add some moderate intensity exercise to your daily routine” may be helpful for them. Encouraging someone who is suffering from compulsive exercising to “add some moderate intensity exercise to your daily routine” would be actively harmful. Since large language models are not medical practitioners, I hesitate to call this “malpractice”, however it is clear that there is an enormous potential for harm even when serving seemingly innocuous, general advice.

And while you may be tempted to address this by including diagnosing as part of the system, that in itself offers extremely high potential for harm. Mental health misdiagnosis can be extremely harmful to patients, even when it happens during consultation with a medical services provider. Adding on the veneer of supposed impartiality from an automated system may increase that harm.

And of course, once diagnosis has been made (even if incorrectly) it then becomes personal identifiable information linked with the patient. However, since an automated system is not actually a clinician, even the limited data privacy protections provided for people in the US by something like HIPAA just doesn’t apply. (In fact it’s a big issue even with online services that use human service providers; Better Help in particular has sold Facebook data from therapy sessions.) As a result, this data can be legally sold by third parties like data brokers.  Even if you don’t intend to sell that data, there’s no way for you to ensure that it never will be, including after your death or if whatever legal entity you use to create the service is bought or goes into bankruptcy.

And this potential secondary use of the data, again even if the diagnosis is incorrect, has an enormous potential to harm individuals. In the US it could be used to raise their insurance rates, have them involuntarily committed or placed under an extremely controlling conservatorship (which may strip them of their right to vote or even allow them to be forcibly sterilized, which is still legal in 31 states). Mental illness is extremely stigmatized and creating a process for linking a diagnosis with an individual has extremely high potential for harm.

Attempting to replicate the services of a mental health clinician through the use of LLMs has the potential to harm the individuals who attempt to use that service. And I think there’s a broader lesson here as well: mental illness and mental health issues are fundamentally not an ML engineering problem. While we may use these tools to support clinicians or service providers or public health workers, we can’t replace them. 

Instead think about what it is specifically you want to do and spend your time doing something that will have an immediate, direct impact. If your goal is to support someone you love, support them directly. If you want to help your broader community, join existing organizations doing the work, like local crisis centers. If you need help, seek it out. If you are in crisis, seek immediate support. If you are merely looking for an interesting engineering problem, however, look elsewhere.