A massive dataset of online images and captions, known as LAION-5B, has been taken down by its creators after a Stanford study exposed that it contained over 3,000 images of child sexual abuse. The dataset was widely used to train AI systems that can generate realistic and explicit images of children, posing a serious threat to their safety and privacy.
Stanford Internet Observatory Study
The Stanford Internet Observatory (SIO), a research group that monitors online threats, discovered that LAION-5B had more than 3,200 images of suspected child abuse, out of which about 1,000 were confirmed by external agencies.
The SIO collaborated with the Canadian Centre for Child Protection and other anti-abuse organizations to identify and report illegal images to law enforcement.
LAION-5B
The SIO’s report, published on Wednesday, also confirmed the rumors circulating on the internet since 2022 that LAION-5B had illicit content. David Thiel, the lead researcher of the SIO, told Ars Technica that “the inclusion of child abuse material in AI model training data teaches tools to associate children in illicit sexual activity and uses known child abuse images to generate new, potentially realistic child abuse content.”
Another report from SIO, in collaboration with the nonprofit online child safety organization Thorn, highlights the swift progress in generative machine learning. This progress enables the generation of realistic imagery that unfortunately contributes to child sexual exploitation through the utilization of open-source AI image generation models.
Thiel’s investigation was prompted by his earlier finding in June that AI image generators were being used to create and distribute thousands of fake but realistic child abuse images on the dark web. He wanted to discover how these AI models, such as Stable Diffusion, a popular text-to-image generator, were trained to produce such disturbing content.
He found out that these models were trained directly on LAION-5B, a public dataset of billions of images scraped from various sources, including mainstream social media websites like Reddit, X, WordPress, and Blogspot, as well as popular adult video sites like XHamster and XVideos. The dataset was created by LAION, a Germany-based nonprofit that aims to advance AI research.
Dataset on hold for supervision
LAION, the nonprofit Large-scale Artificial Intelligence Open Network, told Bloomberg that it has a zero-tolerance policy for illegal content. In an abundance of caution, it has taken down the LAION datasets to ensure they are safe before republishing them.
Google's first edition of the Imagen text-to-image AI tool, initially intended for research purposes, underwent training using a distinct variant of LAION's datasets known as LAION-400M, an earlier iteration compared to the 5B version.
The company clarified that subsequent versions did not rely on LAION datasets. The Stanford report highlighted the findings of Imagen's developers, revealing that the 400M dataset encompassed a broad spectrum of inappropriate content, including pornographic imagery, racist slurs, and harmful social stereotypes.
However, more is needed to solve the problem of the existing models that have already been trained on LAION-5B, such as Stable Diffusion 1.5, which is still widely used to generate explicit imagery.
According to Thiel’s report, the later versions of Stable Diffusion, 2.0 and 2.1, have filtered out some or most of the unsafe content, making it harder to generate explicit content. Still, they have also lost popularity among users.
Stability AI, the British AI startup behind the development and widespread adoption of Stable Diffusion emphasized the company's dedication to preventing AI misuse. The spokesperson stated that the company strictly prohibits using its image models for unlawful activities, including attempts to edit or create CSAM.
“This report focuses on the LAION-5B dataset as a whole,” the spokesperson said. “Stability AI models were trained on a filtered subset of that dataset. In addition, we fine-tuned these models to mitigate residual behaviors.”
Originally published on Interesting Engineering : Original article