Content/trigger warning: Parts of this article may be upsetting to read. I think it is vitally important to call attention to these issues and get the support of legislators to hold companies responsible for proliferating this material. However, it’s a tough read, and some may find it triggering.
There have been some horrible consequences of AI already. False arrests, facial recognition and profiling, and generated pornographic images of celebrities. On top of that, copyrighted work has been used to generate imitations, without the permission of creators, and models have become mountains, using far more data than they should need, leading to an increase in electricity consumption and mining of rare materials. However, there has been a particularly heinous use of AI that may be difficult to hear about. It’s the generation of child sexual abuse materials (CSAM).
Researchers knew it has been possible, despite effort to prevent it, generate realistic CSAM using image generation tools like Stable Diffusion 1.5. Newer versions of Stable Diffusion are better at preventing this, but any image generation AI could be fooled with the right prompt into generating something terrible. Initially, the theory among researchers was that the AI was combining two ideas: sexual content, and young subjects, then combining the two. However, research done by David Thiel for the Stanford Internet Observatory revealed that CSAM is part of the training set. It’s part of what makes the AI work. This has horrific implications. It means that AI can more easily generate realistic CSAM and, most disturbingly, that it may more closely resemble actual victims of CSAM.
AI has had the ability to reinforce stereotypes, but now researchers have shown it could compound trauma. Even when pieces of CSAM is removed from the internet, they could survive in AI models that were built using them, churning out generated images that look similar to real abuse victims.
Tainted Training Data
Thiel’s research found 1,008 validated instances of CSAM, with 3,226 instances of suspected CSAM. There are tools for detecting known CSAM, by hashing the images—turning them into garbled text using an algorithm—and comparing hashes, so no one has to view the disturbing material to confirm that it’s CSAM. Starting with files that LAION already suspected to be “unsafe,” he then found known CSAM using this hashing method. From there, he found similarly-tagged items, and shared it with CSAM classifiers. These are government-ran institutions that add to registries and can identify CSAM without forcing reporters to interact with it. They they can hash them and make it easier to find and report these files in the future.
Unfortunately, Thiel believes the process creates “a significant undercount” of CSAM in the dataset. This is due to limitations in the process, which includes identifying tags and descriptions in other languages, using LAION’s own risk assessments, which obviously aren’t perfect, as well as identifying CSAM that is not yet flagged. He believes there may be much more that he simply wasn’t able to weed out in his research.
Lasting Repercussions
According to Google’s Imagen, who used LAION-400M in early AI research, the dataset included “a wide range of inappropriate content including pornographic imagery, racist slurs, and harmful social stereotypes.” Google, seemingly rightly so, avoided LAION-5B in future AI applications, and has not released models built using LAION publicly. LAION-5B, however, was used in the modeling for Stable Diffusion 1.5. Stable Diffusion now uses better scanning to both prevent using unsafe material during creation of models and block objectionable output better. Its latest versions better protect against misuse, however, many people likely still have 1.5.
“We find that having possession of a LAION‐5B dataset populated even in late 2023 implies the possession of thousands of illegal images—not including all of the intimate imagery published and gathered non‐consensually, the legality of which is more variable by jurisdiction.”
-David Thiel, Stanford Internet Observatory
Stable Diffusion 1.5 is still popular with creators of pornographic content, as newer versions do a better job of preventing this. Thanks to its use of LAION-5B, this could allow it “to generate new and even realistic instances of CSAM,” according to the researchers at Stanford. CSAM will make up a small portion of these datasets, which, while it reduces the variety of realistic CSAM it can generate, it could make images more like the originals that trained the models, depending on the prompts. This means it could look more similar to actual victims of this abusive material.
How to Prevent This
Facebook, Microsoft, and other large tech companies already flag CSAM when they find it and work towards scrubbing it from the net. AI companies have not been so thoughtful with their data. The rush to have the largest models, the most realistic images, means making mistakes with ramifications that will last years. As long as you’re scraping as much as you can from the internet, often without consent, the least you could do is scan for CSAM and help remove it and prosecute those spreading it.
Scanning for CSAM can come in few forms. There’s hash comparison, which uses encoded hashes for comparing potential CSAM with known databases of hashes. This is something most companies already do, and in fact, have to do, legally. Re-scanning your own datasets every few years is also necessary, to catch anything that could have been added to the dataset that was later flagged.
Researchers making AI models could also use age estimation models along with topics to filter out questionable images of young subjects before they make it into models, and help find unreported CSAM.
Once the data is created, any queries should be scanned for possibly bad output, and even generated images should also be scanned to ensure they’re not given to the user if it seems questionable.
This, like so many other issues with AI, could be at least partially solved with proper curation. Curate your data and you can more easily prevent objectionable and heinous output. Social networks need to scan for CSAM, and those laws should apply to anyone generating data models from scraped content as well. No one should be processing this material without legal repercussions.
What Comes Next
Unfortunately, the damage is done. These datasets are in the wild, and anyone using them is knowingly using models built from at some—potentially thousands—of pieces of CSAM. This dataset trained Stable Diffusion 1.5, one of the most well-known and popular image generation services. It’s going to take years for the systems trained on this dataset to expunge this CSAM, and even longer to wipe out use of it elsewhere. Some people are holding on to Stable Diffusion 1.5 specifically to generate objectionable content. They provide plausible deniability to anyone who might have it to generate simulated CSAM. That could make efforts to remove it from usage even more difficult.
This illustrates how important curation is. Yes, it can prevent copyright issues. It can also keep our datasets free from overt racism, bias, sexism, hate speech, and now, we know, even CSAM. These aren’t issues we can easily fix down the road. With models trained on previous output, and the distributed nature of machine learning, fixing these issues later is not going to be a straightforward task. It would be like cleaning up an oil spill, only it occurred in 500 random points in the ocean and keeps spreading.
We need to work now to ensure responsibility for training models based on CSAM and other objectionable content. We need to pass laws that protect copyright and prevent the usage of content taken without permission in AI. We need to force documentation of all data used in training AI. And we need to consider ensuring companies are not knowingly reinforcing bias with the data they choose to ingest.
We need to stop treating this as a problem that will evolve and go away over time, because lives are affected by it now.
Sources:
- Ashley Belanger, Ars Technica
- Emilia David, The Verge, [2]
- Kris Holt, Engadget
- David Thiel, “Identifying and Eliminating CSAM in Generative ML Training Data and Models,” Stanford
- Kyle Wiggers, Devin Coldewey, TechCrunch