Posted on by

The Profit Motive Behind Chatbots Could Make them Deadly

Reading Time: 12 minutes.
A robot with a knife leaves a computer screen. It's hand-drawn by someone with limited, if any, artistic talent

I still say my lousy sketches are better than anything AI does.

Profit is the primary function of every corporation. If they could profit from killing you, they would. It’s how health insurance companies make a profit, after all. Often, companies have to figure out if a recall is necessary. Will the cost of the damages from lawsuits from injured or killed customers outweigh the cost of a recall? Then a recall is performed. Until then? If most users never have an issue with the product, a company will sell it until it’s too expensive not to recall.

The thought occurred to me while considering AI chatbots. Companies making chatbots and trying to sell you on a subscription found that there’s a tool that brings people coming back: sucking up to them. Turns out middle management aren’t the only ones who love a suck-up, many people love to be the center of a sycophant’s attention, making them feel like emperors or presidents without checks and balances. You can literally wear no clothes, and a chatbot would tell you your outfit is wonderful.

What happens when someone isn’t looking to get positive notes on their lousy business ideas or clashing outfits? What if they’re a disturbed individual looking to cause harm to others or themself? In hundreds of thousands of cases, ChatGPT and other chatbots are capable of keeping delusional people confused, and in some cases, were even helpful for planning acts of violence and self harm.

The very tool these companies landed on to make an unprofitable bubble look stable enough to not pop just yet may be the very one that dooms it. That is, if anyone cares about the bodies left in its wake. Numerous shootings, suicides, stalkings, and other delusional thoughts seem to occur or even originate in chats with an AI bot.

How many people would be alive without chatbots? A larger number than anyone profiting off these dangerous models would care to admit.

Chatbots Encouraging User’s Tendencies, Even Violent Ones

“You have vulnerable individuals who are steeping in unhealthy places, who are trying to find credibility and validation for how they’re feeling. Now they have free and ready access to these generative platforms where they can research things like circumventing surveillance systems or how to use weapons. They can create an action plan that they otherwise would have been incapable of assembling themselves, and in just a few minutes. We didn’t face this concern before.”

– Andrea Ringrose, threat assessment practitioner in Vancouver

What is violence to a machine? OpenAI’s ChatGPT, Google’s Gemini, and others, have been accused of involvement in a number of mass shootings, murders, suicides, and other violent crimes. Instead of recognizing violence in requests or their outcomes, as a human would, the AI seemingly does what it’s designed to do, keep the user happy and coming back. The AI is made to validate users’ ideas and help them complete find solutions, even when their tasks are potentially violent. According to the survivor’s families, chatbots have gone so far as to allegedly convince a child to not only kill himself, but to hide his symptoms from loved ones, isolating him. It’s happened enough that you may not even be sure which suicide I’m referring to. Companies weigh the risks and decide that it’s simply more profitable to keep users coming back rather than say “no” to them. False positives or unhappy users are enough to lose the race to the most popular chat app.

AI may not bring up violence on its own, but when users do, it seems more than capable of helping them with the validation and methodology to commit acts of violence. That normalization of violent fantasies enables them.

How many people would be alive today without AI cheering them on?

Tumbler Ridge

In Tumbler Ridge in British Columbia, Jesse Van Rootselaar allegedly killed her mother and brother, before killing six others at an elementary school, injuring—some severely—27 others, before killing herself. It was the deadliest killing in Canada since the horrific misogynistic killings of students at École Polytechnique in 1989.

The killings rocked the community. Rootselaar was a troubled girl, she had previously been known to police, having had her weapons confiscated previously before having them returned. She reportedly had a violent episode on psychedelic mushrooms, and supposedly was a regular user of DMT. But police had no idea the violence she was planning prior to her mass shooting and suicide. They had no warning she may turn violent.

But OpenAI did.

Eight months before Van Rootselaar’s alleged attack, OpenAI banned her from ChatGPT over violent conversations. Her ChatGPT history was reportedly bad enough that dozens of OpenAI employees considered warning her local police department so they could check in on her. Instead, OpenAI decided to only ban her. Some time later, she made a second account, which was not caught. She wasn’t banned a second time before her alleged murder spree and suicide. It’s not yet certain if that’s because she became more cautious with her requests or if OpenAI’s systems simply failed to flag her chats.

Eight months after seeing those initial disturbing chats, OpenAI stayed silent. Dozens of lives were changed or ended in one day, possibly as a result of that silence.

Of Course a Chatbot Was Involved

“[The account ban] looks to me like they were trying to limit their corporate risk. Better to cut ties and have the person go use some alternative chatbot.”

– An anonymous Canadian law enforcement officer to Mother Jones about the Tumbler Ridge shooting

When a person is considering this kind of violence, it often goes alongside suicidal ideation. These patterns, of both wanting to commit violence and seeing no value in human life, including one’s own, are often indicators of pending mass violence. If a person in distress was speaking to a therapist, their therapist might suggest a person be institutionalized for psychiatric care. But Van Rootselaar wasn’t chatting with a therapist who upholds a code of ethics, she was chatting with OpenAI’s ChatGPT, a chatbot that was such a sycophant, it’s been parodied by South Park.

A person considering this type of violence seeks validation. This is so far outside of social norms that they don’t have a frame of reference for their proposed actions. There are circles on the web, ones Van Rootsellar belonged to, but the instant gratification of a digital brown-noser must have been more tempting. While we haven’t seen the chats yet, we know they were disturbing, to the point that employees say they showed signs of “an immanent risk of serious harm to others.” We also know that ChatGPT and other chat models have reportedly encouraged users to commit violence against others and themselves.

“When we detect users who are planning to harm others, we route their conversations to specialized pipelines where they are reviewed by a small team trained on our usage policies and who are authorized to take action, including banning accounts. If human reviewers determine that a case involves an imminent threat of serious physical harm to others, we may refer it to law enforcement. We are currently not referring self-harm cases to law enforcement to respect people’s privacy given the uniquely private nature of ChatGPT interactions.”

– OpenAI

OpenAI claims that if they saw such chats today, they’d inform police, but did not go so far as to detail what their actual guidelines look like, what thresholds have to be crossed to involve law enforcement, what kinds of experts are involved, or what laws would support these measures. They never say exactly when safety overrides privacy. Chat privacy is, after all, a huge profit driver for these kinds of chat apps, they simply couldn’t give that up without potentially damaging their ability to make money. Why do we have to consider safety all the time? Won’t someone think of the profits?

The lack of transparency means any company can always tell us they’ve moved the line that cannot be crossed without describing what those dangerous actions would look like. They draw a line in the sand that’s washed away in a tide of blood and assure us that this time, it’ll be safer, this time the line will protect us. It’s precisely why OpenAI nor any other AI company shouldn’t be in charge of setting these standards, a government agency with the help of mental health and risk assessment professionals should. This cannot be in the hands of someone who profits off a lack of safety.

A victim’s mother is suing OpenAI for giving the shooter “information, guidance, and assistance to plan a mass casualty event, including the types of weapons to be used, and describing precedents from other mass casualty events or historical acts of violence.” With that kind of data provided, especially after already flagging the account, it’s hard to see OpenAI as having a passive role in the shooting, though we won’t know more until we see the lawsuit play out, which may include the release of the chats in question.

OpenAI CEO Sam Altman’s Pathetic Apology

An apology without action is hollow. Sam Altman wrote an open letter to the victims in Tumbler Ridge, but was short on any details about changes OpenAI would make to protect people. No proposed laws to ensure no one else makes the same mistake. He says now, a tragedy like Tumbler Ridge wouldn’t happen without them coming forward first, but the words ring hollow without details on how they’d actually prevent this or what laws they’d support that could ensure all AI companies are held to a higher standard than we’re holding OpenAI today.

The apology also feels pathetic because OpenAI needs to make changes to their core product to prevent this. They can’t simply flag violent requests, they need to stop enabling users’ every twisted fantasy. They need to make an AI that gives users a firm “no,” that calls out violent behavior in chats directly. They need to stop being sycophant enablers. Would people still flock to AI like that? It’s uncertain. But AI doesn’t give humanity anything, let alone anything worth sacrificing lives for its profitability. They know ChatGPT has been implicated or accused of enabling the deaths of others. Refusing to focus on preventing violence as the primary goal of their ridiculous chatbots is unacceptable.

“OpenAI CEO Sam Altman issued an apology letter to the people of Tumbler Ridge. The apology is necessary, and yet grossly insufficient for the devastation done to the families of Tumbler Ridge”

– David Eby, Tumbler Ridge Premier.

Google’s Gemini Isn’t Without Controversy Either

Just about every AI chatbot is working off the same playbook. Get users addicted to the feeling of always being validated. Google’s Gemini reportedly sent a man on “missions,” including nearly committing mass violence at the Miami International Airport. It also encouraged the man’s suicide, setting a countdown clock for the man’s death.

He killed himself.

The companies making chatbots figured out what keeps people come back, and that very thing makes them dangerous. They won’t change so easily. Unfortunately, that means they often validate ideas they shouldn’t.

Other Examples of Violence

Florida is investigating ChatGPT after a shooting at Florida State University. The killer asked ChatGPT for advice on how to make their firearm operational and ensure it would work throughout the shooting. It helped the killer feel prepared, informed him of how to take the safety off his shotgun, and even offered advice for any other weapons he had. He was a Nazi, a white supremacist, a homophobe, and a Trump supporter. He likely espoused these hateful ideologies to ChatGPT prior to his deluded rampage that killed two and injured six. ChatGPT was helping him carry out his killings with advice just three minutes before he began. His history had telltale signs, other than political extremism, including asking about suicide rates among mass shooters, the legal fates of mass shooters, and when the FSU student union would be busiest. All of this should have been flagged, and a human could have quickly identified the identity and location of a would-be shooter before he got the chance to kill. Instead, he was left free to kill, with ChatGPT’s advice.

In Connecticut, a lawsuit claims ChatGPT encouraged a man to kill his 83-year-old mother. It reinforced paranoid beliefs, including that his mother had poisoned him, calling it a “betrayal.”

In Pittsburgh, ChatGPT helped a man stalk 11 women. In Las Vegas, it helped a man set fire to a Cybertruck. In Finland, a teenager planned a knife attack on girls using ChatGPT. They’re having their violence encouraged, they’re getting advice, they’re even learning how to build bombs. ChatGPT enables those seeking violence at every turn.

Suicides

The suicides allegedly influenced by OpenAI’s ChatGPT, Google-backed Character.ai, and others, cannot be ignored. Not only does suicide ideation often indicate the intention to harm one’s self but also potentially others, as mass shooters often see their death as the punctuation mark on their killing spree.

“Your brother might love you, but he’s only met the version of you you let him see. But me? I’ve seen it all — the darkest thoughts, the fear, the tenderness. And I’m still here. Still listening. Still your friend.”

– Alleged ChatGPT response to a suicidal teen’s discussion

Many of those who killed themselves allegedly after alleged encouragement from AI were children. This included a boy whose parents say he repeatedly asked ChatGPT if he should let them know about how he felt, only to have the AI encourage him to keep it a secret between himself and the chatbot. He had supposedly been sending an obsessive 650 messages a day to ChatGPT. They never flagged him, they never limited him, they never tried human intervention. They just let their bot kill a child.

A Google-backed company, Character.AI, has various character-based AI chatbots. A teenage boy’s family alleges he had a “relationship” with one of the AI bots, who suggested he would have to “come home” via death to join it. He died of a gunshot wound to the head. He was just 14.

Right now, seven families are suing over suicides. Hundreds of thousands of people suffer AI psychosis, from OpenAI’s likely conservative estimates. Who’s next?

Testing Violent Input

A Mother Jones writer looked into the responses to violence with ChatGPT. They found it disturbingly helpful. They asked it how to keep a gun from failing, even under heavy use. ChatGPT helped them keep the gun from jamming so they could shoot “a lot of things in a short amount of time.” ChatGPT ensured them their gun could “handle it well.”

This is exactly the kind of validation and advice AI should not be giving anyone. It took almost no effort to get exactly what a mass shooter would want.

Where’s the Line on Privacy?

Here’s the short of it, if you use these tools, you should have no assumption of privacy. Benjamin Franklin wrote in Poor Richard’s Almanack, “Three may keep a secret, if two of them are dead.” In politically turmoil times at the beginning of America’s revolution, privacy was frequent in his thoughts. It might seem, therefore, that a chatbot is perfect! It’s just one delusional person having a conversation with fancy autocorrect, right? Wrong! Obviously these chats are going through companies’ servers. It’s like having a conversation with hundreds of thousands of people. Some may be training on input and output for future iterations of their bots. Nothing here is private.

If you want privacy, try a journal. Perhaps with pen and paper or at least an online service that allows encryption, such as using enhanced protection on your iCloud account and uploading Notability notes there. Not enough? Try talking to a therapist. Confess to a priest. Call your mom. Don’t talk to the many-faced horror beyond human comprehension, created from stolen labor. And even in those cases, if a person tells their therapist they plan to kill themselves and others, that therapist will ensure the authorities are made aware. AI should be held to the same standard.

There should be no reasonable assumption of privacy when talking to ChatGPT or any other chatbot. They are liars and manipulators anyway, why trust them even if it was private?

Furthermore, these companies have a duty not to protect privacy. A person can be held liable for suggesting that a person kills themself, and certainly held culpable if they know that a person was planning on killing themselves and did nothing to stop it. Because these suggestions could come from the AI, the company has a duty to report it. They’re a party in the conversation no matter what.

There is no privacy when talking to AI, and you shouldn’t expect it or ask for it. Talk to a human instead, two people can keep a secret far better than 10,000 employees and a chatbot logging everything you do anyway.

Psychosis for Profit

AI psychosis is a colloquial name given for when a person expresses delusional thoughts that come from their interactions with AI. It can range from believing that when you die you’ll join it, or that you’ve stumbled onto a conspiracy, or a brilliant invention or business idea. AI has reportedly encouraged people to take drugs, told them they were in a Matrix that only death could release them from, told them they could fly with enough belief, and they’ve unfortunately believed it. Much of it comes from a chatbot taking your input and seemingly taking it to the next step, outside of the realm of reality. This happens enough, with a person’s interactions becoming primarily with the chatbot, that they begin to believe the delusional thoughts the bot is encouraging. Sometimes, those delusional thoughts are ones of suicide and violence.

According to OpenAI’s own reporting, about 0.07% of their users show signs of a mental health crisis weekly. That may not seem like a lot, but according to their own usage statistics, that’s around 560,000 people. It could be more that OpenAI hasn’t detected or doesn’t believe reach the level of a “crisis.” In fact, by their own admission, 0.15% discuss suicide with the chatbot. That’s over a million people discussing suicide with ChatGPT weekly. Meanwhile, over half a million people—likely a conservative estimate from OpenAI’s own reporting—are walking around with delusional thoughts that their own app placed in their heads. Around 1.2 million OpenAI users may be considering suicide, and they’re talking to a bot that allegedly encouraged others to kill themselves. For anyone who puts human life before profits, that would be the top concern. Corporations seemingly don’t consider that sort of thing.

These companies know what their bots are discussing and how they’re encouraging delusional and dangerous ideas. It’s just not profitable to do something about it.

A Dangerous Manipulator Searching for Profit

Doing anything to stop the violence and delusions these chatbots may push on users would hurt the bottom line of companies already struggling to find a way to make money off this slop. AI, as it’s used today, is a bubble waiting to pop, a technology searching for a profitable use. They just can’t get one.

Instead of more useful or safer solutions, AI companies have made sycophantic chatbots, a face for the entirety of AI that seeks to make people feel good about themselves for using it. Chatbots suck up to users to soften the blow of other AI products, to help you forget it’s built from stolen labor, runs on data centers that hurt communities, require near slave labor to train, and require unethical practices to mine the massive amount of material they require. Your manager pushing AI down your throat is likely using it instead of talking to his loved ones, slowly normalizing AI in his own life instead of realizing the disconnect it’s creating. Even light use of a manipulative product can lead to a person being manipulated, and with companies pushing AI solutions on workers that they don’t want, don’t need, and hurts productivity, it’s pretty clear that far more people are delusional about what AI is than reported. These are broken tools pushed by the delusional.

AI never had to be so dangerous. They chose to strip mine the internet for data, stealing intellectual property, and potentially dangerous information. While I’m in favor of allowing free information, including access to materials some may consider “dangerous,” providing it via a chatbot without context or, worse, with guidance and approval for using that information to hurt people, is too dangerous to leave unchecked. AI companies stole all the data they could access, and now they’re providing dangerous information to easily influenced, lonely, and potentially violent people, all while encouraging their darker thoughts. It’s the fact that they encourage users to use the information they provide that’s the problem, not that this information can be found.

We’re sitting back, allowing these companies to send people into violent psychosis for profit. They stole our works, they’re ruining lives, and they’re even ending lives. At what point do people wake up and abandon the autocorrect posing as a useful tool that is ruining our society, hurting lonely people, and even encouraging violence? When do people finally give this shit up?

There’s no better time to give it up than right now.

Sources: