{"id":71284,"date":"2024-05-27T17:02:10","date_gmt":"2024-05-27T17:02:10","guid":{"rendered":"https:\/\/www.humanium.org\/en\/?p=71284"},"modified":"2025-12-13T20:14:22","modified_gmt":"2025-12-13T20:14:22","slug":"combating-the-rise-of-ai-generated-child-sexual-abuse-material","status":"publish","type":"post","link":"https:\/\/www.humanium.org\/en\/combating-the-rise-of-ai-generated-child-sexual-abuse-material\/","title":{"rendered":"Combating the rise of AI-generated child sexual abuse material"},"content":{"rendered":"\n

The world is undergoing a boom in accessible <\/strong>artificial intelligence<\/strong><\/a> (AI) technologies. New tools and capabilities present novel opportunities for criminal and malicious actors to exploit children. As lawmakers, policymakers and law enforcement work to keep up with criminal innovations, children are growing increasingly susceptible to AI-generated child sexual <\/strong>abuse<\/strong><\/a> material (CSAM). New technologies are enabling criminals to create, amplify and distribute CSAM at an unprecedented rate. Without urgent intervention, this trend threatens to undermine the <\/strong>safety<\/strong><\/a> of children online. <\/strong><\/p>\n\n\n\n

What is AI-generated CSAM?<\/strong><\/h2>\n\n\n\n

AI-generated CSAM is synthetic or life-life CSAM created using AI tools (Clark, 2023). This content poses a dual threat: in and of itself it allows for the generation of new and harmful CSAM, and simultaneously AI tools allow criminals to access and disseminate other CSAM at a faster rate than ever before. Illicit content can contain altered images of real children or fraudulent creations. AI tools\u2019 capability to create content that did not previously exist is often termed \u2018generative\u2019, and this modality widens the net of potential CSAM victims given the images are fraudulently created. <\/p>\n\n\n\n

AI has presented an unprecedented opportunity for criminals to create an unlimited volume of CSAM. All forms of AI-generated CSAM are illegal, irrespective of whether they depict or reflect a real child or not. AI tools are sophisticated, that most AI-generated CSAM is \u2018visually indistinguishable\u2019 from real CSAM, including for trained analysts (Internet Watch Foundation, 2023).<\/p>\n\n\n\n

Trends and prevalence<\/strong><\/h2>\n\n\n\n

Though the use of AI to generate and propagate CSAM is relatively novel, there are emerging insights about the methodologies being employed by criminal actors. In terms of generation, open access AI software such as Stable Diffusion is being used to generate fraudulent CSAM content (Crawford and Smith, 2023). <\/p>\n\n\n\n

The ease of this and other software allows individuals \u2013 who do not necessarily have prior technological experience \u2013 to create new images based on word prompts and simple instructions (Crawford and Smith, 2023). The text-to-image technology is quick, accurate and allows large volumes of content to be created simultaneously (Internet Watch Foundation, 2023). <\/p>\n\n\n\n

In terms of housing and disseminating CSAM, content-sharing sites such as Patreon are being used to store AI-generated CSAM images (Crawford and Smith, 2023). This is enabling criminals to purchase subscriptions on the same services to access images consistently (Crawford and Smith, 2023). Patreon have expressed their zero tolerance towards this content appearing on their website, however identifying and sanctioning actors who are exploiting these platforms is easier said than done (Crawford and Smith, 2023). <\/p>\n\n\n\n

Global struggles against AI-enhanced online child abuse<\/strong><\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n

This challenge is not limited to Patreon; the United Kingdom\u2019s<\/a> (UK) National Crime Agency (NCA) believes that more end-to-end encryption \u2013 the process of securing data between two communicating parties and preventing third parties from accessing it \u2013 within Meta\u2019s platforms (Facebook, Instagram and others) would mean law enforcement would lose access to between 85 and 92% of CSAM referrals coming through the platforms (Home Office, 2023).<\/p>\n\n\n\n

Studies in the US suggest Facebook, Instagram and WhatsApp are the social media platforms housing the greatest proportion of illicit material (McQue, 2024). These are quickly followed by other large scale social media entities, reflecting the central role of some of the world\u2019s most popular websites and applications in enabling CSAM dissemination. <\/p>\n\n\n\n

Owing to the ease of production of CSAM facilitated by AI, it is difficult to accurately measure the scale and prevalence of the content online. The UK\u2019s Home Secretary stated that 800 arrests have been made of sexual predators and the government safeguard 1,200 children per month (Home Office, 2023), but this is not limited to the online domain. Further, AI software can be downloaded and run online, this means malicious actors can create limitless content whilst disconnected from the internet, obscuring them from law enforcement (Internet Watch Foundation, 2023). <\/p>\n\n\n\n

The limited analysis that does exist points to an alarming volume of AI-generated CSAM being produced in a short space of time. Analysis in the UK found 20,254 images had been posted to just one AI CSAM forum in the month of September 2023 (Internet Watch Foundation, 2023). <\/p>\n\n\n\n

In the United States of America<\/a> (USA), the National Center for Missing and Exploited Children (NCMEC) only started receiving AI-generated CSAM in 2023, and have already received reports linked to nearly 5,000 pieces of content (McQue, 2024). In 2023, the USA\u2019s Cyber Tipline received nearly 36 million reports referring to CSAM incidents, over 90% of the content was uploaded on foreign soil and over 60,000 of the reports involved a child in perceived immediate danger (McQue, 2024). <\/p>\n\n\n\n

Law enforcement and policy responses<\/strong><\/h2>\n\n\n\n

Given the cross-cutting nature of the AI-generated CSAM challenge, multi-sectoral approaches are required to meaningfully disrupt the trend. In the private sector 27 organisations, including TikTok and Snapchat among others, recently partnered with several governments, including the USA<\/a>, Germany and Australia<\/a> to explore new ways to combat AI-generated CSAM (Clark, 2023). <\/p>\n\n\n\n

Bilateral and multinational government interventions are also beginning to surface: in 2023 the USA and the UK issued a joint statement reaffirming their commitment to combatting generative AI (Home Office and US Secretary of Homeland Security, 2023). This was supported by a landmark UK government bill, the Online Safety Bill, which imposed stricter requirements on private sector entities to do all they can to identify and remove CSAM content from their websites (Home Office, 2023). <\/p>\n\n\n\n

Strategic responses to AI-generated CSAM<\/strong><\/h2>\n\n\n\n

The growing threat of AI-generated CSAM necessitates a prompt and comprehensive response. Among other interventions, there are a few key things that governments must do to catalyse a robust response.<\/p>\n\n\n\n