According to Futurism, the police department in Heber City, Utah, was forced to explain why an AI-generated police report claimed an officer transformed into a frog. The incident occurred in 2024 while testing Draft One, an AI software from Taser-maker Axon that uses OpenAI’s GPT models to write reports from bodycam audio. The software hallucinated the fantastical ending after picking up background audio from the Disney movie ‘The Princess and the Frog.’ Despite the glaring error during a mock traffic stop, Sergeant Rick Keel said the tool saves him six to eight hours weekly and is user-friendly. The department is still deciding whether to keep using Draft One and is also testing a competing AI software called Code Four, released earlier this year.
The Hallucination Problem
Here’s the thing: this isn’t just a funny story. It’s a perfect, almost poetic, example of the core problem with generative AI—its tendency to confidently make stuff up. The software heard a movie title and wove it into the official narrative of a police encounter. That’s terrifying when you consider the weight a police report carries in the justice system. Experts, like American University law professor Andrew Ferguson, warned about this exact scenario last year, fearing the ease of the tech would make officers “less careful with their writing.” And look what happened. The fundamental issue of AI hallucinations is well-documented, but deploying it in a high-stakes, real-world context like law enforcement? That’s a whole other level of risk.
Bias and a Lack of Transparency
But the frog fable is just the tip of the iceberg. Critics have much deeper concerns. Generative AI models are known to perpetuate existing racial and gender biases, which is especially alarming given law enforcement’s historical record. Then there’s the transparency issue—or complete lack thereof. A recent investigation by the Electronic Frontier Foundation found that with Draft One, it’s often impossible to tell what parts of a report were written by AI and what parts were written by an officer. They argue the system is “deliberately designed to avoid audits,” which could introduce deniability and reduce officer accountability. So you get a combo platter: potential bias baked into the narrative, and no clear way to audit where that narrative came from. What could go wrong?
The Rush to Automate Policing
So why are departments rushing to adopt this? The pitch is simple: efficiency. Less paperwork. Sergeant Keel’s quote about saving six to eight hours a week is the entire sales pitch. And look, I get it. Bureaucratic drag is real. But this feels like a classic case of solving one problem by creating several much bigger ones. Axon, a company that profits immensely from selling tech to police, is now providing the tools that could shape the very facts of a case. As one critic told the AP, that relationship alone is alarming. The promise is a revolution, but the beneficiaries seem unclear. Will it be the public seeking justice, or the departments seeking cost savings and procedural cover? The broader implications for the criminal justice system are massive and largely untested.
A Reality Check for AI Adoption
Basically, this frog story is a reality check. It’s a slapstick demonstration of a deeply serious flaw. When you integrate powerful but imperfect AI into critical infrastructure, you’re gambling with outcomes. This isn’t a chatbot giving bad recipe advice. This is about official records that determine guilt, innocence, and liberty. The Heber City department is now at a crossroads, testing another AI tool. But the core issue remains the same: any system that can’t distinguish between a real police interaction and a Disney musical has no business drafting legal documents. The pushback is growing, and it should. In fields where accuracy and accountability are non-negotiable, maybe the best tech solution isn’t the flashiest one. Sometimes, the most reliable tool is still a human being—flawed, yes, but at least capable of knowing they aren’t an amphibian.
