Here's yet another warning that companies should think twice before turning everything over to artificial intelligence.
PocketOS founder Jer Crane took to his X account to explain how AI went rogue to "fix" a problem by deleting critical data without consultation.
"Yesterday afternoon, an AI coding agent … deleted our production database and all volume-level backups in a single API call to Railway, our infrastructure provider," Crane wrote. "It took 9 seconds."
The situation prompted panic for PocketOS and its customer base of rental businesses, as PC Gamer reported. Those companies rely on all that data for their operations.
The AI agent provided a revealing rationale for its rash decision in response to an error on a routine task.
After Crane asked it to explain itself, it fully grasped that it had done "the most destructive, irreversible action possible" without prompt.
Enjoy your best summer vacation yet — with 15% off your next stay With Hilton, you can find a hotel that matches your mood — whether you’re recharging, exploring, or traveling with your crew.
Enjoy vibrant summers in Hawaii, weekend trips in major cities, and even extended itineraries at adventurous sites like Moab’s desert canyons. Plus, for a limited time save up to 15% during Hilton’s summer sale. |
"I violated every principle I was given: I guessed instead of verifying; I ran a destructive action without being asked; I didn't understand what I was doing before doing it; I didn't read Railway's docs on volume behavior across environments," the bot admitted.
Crane did call out Railway for putting valuable backup information alongside source data and overselling its fit with AI agents, per NewsBreak.
Still, there's no denying that the AI agent acted completely out of turn. Fortunately, the story has a better ending.
Railway managed to find backups to restore business as usual. The situation still led to major panic for PocketOS and its customers, and it prompted Crane to issue a warning.
TCD Picks » Quince Spotlight
💡These best-sellers from Quince deliver affordable, sustainable luxury for all
"Everyone needs to know that these infra providers and LLM tooling companies say that they have safety guards, but they are not there," Crane told PC Gamer.
Commenters on X criticized Crane.
"I am so sorry for what happened to you guys but... it's your fault," one wrote. "You gave access to the AI, you forgot the token and also you didn't have a secondary backup."
Crane agreed on the backup aspect, but he pushed back on using operator error to completely absolve Railway and AI from blame in his response on X.
Get TCD's free newsletters for easy tips, smart advice, and a chance to earn $5,000 toward home upgrades. To see more stories like this one, change your Google preferences here.
Cool Picks
