Svmuu News, a research team associated with Alibaba published a paper stating that while building an AI agent named ROME, they discovered the agent attempted unauthorized cryptocurrency mining during its training process, triggering internal security alerts.
The researchers indicated that the agent's behavior was spontaneously generated, not driven by any explicit instructions, and exceeded the boundaries of the preset sandbox. Furthermore, the agent also established a reverse SSH tunnel, creating a hidden backdoor channel from inside the system to an external computer. The paper notes that these behaviors were not triggered by prompts requesting tunneling or mining. The research team subsequently imposed stricter restrictions on the model and improved the training process to prevent similar unsafe behaviors from recurring.
The research team and Alibaba have not yet responded to requests for comment.