A single cloud compromise can fuel an army of AI sex bots

An anonymous reader quotes a report from KrebsOnSecurity: Organizations that lose credentials to their cloud environment could quickly become part of a disturbing new trend: cybercriminals using stolen cloud credentials to exploit and resell sexualized AI-powered chat services. Researchers say these illegal chatbots, which use modified jailbreaks to bypass content filtering, often devolve into dark role-playing scenarios, including child sexual exploitation and rape. Researchers at security firm Permiso Security say attacks on generative artificial intelligence (AI) infrastructure such as Amazon Web Services’ (AWS) Bedrock have increased significantly over the past six months, especially when someone in the organization accidentally exposes their cloud data or key online, like like in a code repository like GitHub.

While investigating the misuse of AWS accounts for several organizations, Permiso discovered that attackers had used stolen AWS credentials to interact with the large language models (LLMs) available on Bedrock. But they also quickly discovered that none of these AWS users had logging enabled (it is disabled by default), leaving them with no insight into what attackers were doing with that access. So Permiso researchers decided to leak their own test AWS key on GitHub, while turning on logging, so they could see exactly what an attacker might ask for, and what the responses might be. Within minutes, their bait key was picked up and used in a service that offers AI-powered online sex chats.

“After reviewing the prompts and responses, it became clear that the attacker was hosting an AI role-playing service that uses common jailbreak techniques to allow the models to accept and respond to content that would normally be blocked,” Permiso researchers wrote in a report released today. “Almost all of the roleplay was of a sexual nature, with some of the content straying into dark topics such as child sexual abuse,” they continued. “Over the course of two days, we saw more than 75,000 successful model calls, almost all of them sexual in nature.”