Here's yet another warning that companies should think twice before turning everything over to artificial intelligence.
PocketOS founder Jer Crane took to his X account to explain how AI went rogue to "fix" a problem by deleting critical data without consultation.
"Yesterday afternoon, an AI coding agent … deleted our production database and all volume-level backups in a single API call to Railway, our infrastructure provider," Crane wrote. "It took 9 seconds."
The situation prompted panic for PocketOS and its customer base of rental businesses, as PC Gamer reported. Those companies rely on all that data for their operations.
The AI agent provided a revealing rationale for its rash decision in response to an error on a routine task.
After Crane asked it to explain itself, it fully grasped that it had done "the most destructive, irreversible action possible" without prompt.
Support pets in need with these special-edition memory foam shoes BOBS from Skechers has helped over 2 million shelter pets around the world — and the charity program just announced this year’s Paws for a Cause design-winning sneakers. These "hound huggers" and "kitten kicks" sneakers are machine washable and equipped with memory foam insoles. Plus, they were designed by passionate students who were inspired by their very own rescue pets. BOBS from Skechers is also committed to donating half a million dollars to the Best Friends Animal Society this year to help every dog and cat experience the safety and support of a loving home. |
"I violated every principle I was given: I guessed instead of verifying; I ran a destructive action without being asked; I didn't understand what I was doing before doing it; I didn't read Railway's docs on volume behavior across environments," the bot admitted.
Crane did call out Railway for putting valuable backup information alongside source data and overselling its fit with AI agents, per NewsBreak.
Still, there's no denying that the AI agent acted completely out of turn. Fortunately, the story has a better ending.
Railway managed to find backups to restore business as usual. The situation still led to major panic for PocketOS and its customers, and it prompted Crane to issue a warning.
TCD Picks » Quince Spotlight
💡These best-sellers from Quince deliver affordable, sustainable luxury for all
"Everyone needs to know that these infra providers and LLM tooling companies say that they have safety guards, but they are not there," Crane told PC Gamer.
Commenters on X criticized Crane.
"I am so sorry for what happened to you guys but... it's your fault," one wrote. "You gave access to the AI, you forgot the token and also you didn't have a secondary backup."
Crane agreed on the backup aspect, but he pushed back on using operator error to completely absolve Railway and AI from blame in his response on X.
Get TCD's free newsletters for easy tips, smart advice, and a chance to earn $5,000 toward home upgrades. To see more stories like this one, change your Google preferences here.
Cool Picks
