AI is driving me absolutely f-in crazy. I teach college students (both grad and undergrads) and the majority of the assignments they turn in are a cut/paste of ChatGPT nonsense. When I open my enterprise gmail account and start to write an email, Gemini populates a completely random message that has nothing to do with my current task. Earlier this week, I needed to modify my resume into a one-page bio for a conference and even though I have tons of publications and uploaded a word doc with my resume…AI made up a paper that I didn’t write!
Making up a fake citation is a high crime in academia that will get someone blacklisted for life. Unfortunately, many students don’t realize this because I’ve been pitched a thesis using a social theory that doesn’t exist and another student cited a paper that I authored with an AI-written quote that wasn’t in my paper.
The entire US economy is riding on the dream that an AI technological revolution that rivals the invention of electricity or telephones is about to happen. Tech CEO hype men like Elon and Sam Altman are claiming AI is the most important invention in the history of humanity. At the same time, I’m pretty sure the Anthropic Claude model that’s connected to the US military’s top secret intelligence files accidentally bombed an elementary school in Iran.
The current batch of AI models suck at doing real world tasks accurately because they are operating exactly the way they are designed to operate. If you treat an LLM like a calculator that is occasionally “broken”, you are misunderstanding the machine.
When an LLM gets simple math wrong, it’s because it’s trying to “read” the answer out of its probability cloud rather than “calculating” it. To “fix” this, you’d have to strip away the fluid neural network and replace it with rigid code, which would destroy the AI’s ability to summarize your meeting notes, create a funny meme, or write poetry.
This is the same reason why ChatGPT can’t complete the simple task of filling out a March Madness bracket. Even when I gave the instructions to pick the winners round by round, ChatGPT matched up teams that can’t play against each other based on the binary logic rules of a bracket tournament (red arrows). For example, ChatGPT’s first round pick is Houston over Clemson followed by Houston over Nebraska in the second round. It’s impossible for Houston to beat Clemson because Clemson is playing Iowa during the first round. Houston can’t play Nebraska in the second game because they can’t meet until the Elite 8 based on the structure of the South region.
ChatGPT did this because instead of following a strict logic rule of “teams must play against their pre-selected opponent”, ChatGPT followed the general theme of picking teams in the correct tournament regions with its own set of rules for making the game by game matchups.
ChatGPT, Gemini, Claude, DeepSeek, GROK, and all the other LLMs are stochastic (probabilistic) engines. When the model makes an “error”, this happens because the program is running exactly as designed. These mistakes are due to the structural paradox of LLMs because the exact architectural features that make them feel “magical” are the same ones that make them dangerous in high-stakes environments like academia, critical business processes, or integration into DOD weapons and classified intelligence systems.
A book has words on pages and if you cite the paragraph on page 221, those words will be exactly the same for anyone who reads your citation or opens the same edition of the book to that page. Web search is very similar because if you find text on a website, a search for that exact text will bring you back to the same place (with some variation because websites are editable). The internet was basically a giant library of mostly static text information in predictable, storable, and searchable places. Before AI search summaries, search engines were essentially high-speed librarians.
Unlike a google search, a LLM chatbot like ChatGPT doesn’t have a library of knowledge to directly reference. To train an AI model, billions of files from the internet are converted into a giant map of statistical relationships (vectors) between words. This process makes an LLM into a multidimensional map of “word neighborhoods”. Because an LLM doesn’t find the exact word or paragraph to cite, it calculates the most likely next value in a sentence from the neighborhood of surrounding words.
When AI gives you a fake answer, this "hallucination" is not a failure of the LLM to find the right passage in the book because it doesn’t have any books, just a jumble of numbers. Turning words into statistical relationships allows the AI to be creative and capable of synthesizing and producing new information. That “creativity” comes from the ability to jump to the next most likely word without a grounding in rigid, external reality.
You cannot have the emergent creativity of an LLM without the inherent risk of fabrication. To “fix” hallucinations would require turning the LLM back into a rigid, rule-based Google Search that can only find existing text. Because an LLM is calculating likelihood and not truth, a fake citation isn’t an error to the AI because it’s just a statistically plausible string of text. In its “mind”, a fake paper title that sounds like something I would write is a successful output.
To make these models run, they “compress” the entire internet into a set of weights. It’s like turning a high-definition movie into a tiny, blurry JPEG so it fits on a thumb drive. This compression allows the model to “understand” concepts and generalize information across any query. But the compression is “lossy” which means the fine details (like the correct author of a paper or the exact wording of a quote) get dropped from the outputs.
If an LLM remembered every detail perfectly from 11 trillion pieces of data scraped from the internet, it would be too large to run and too rigid to be “intelligent.” The magic of AI comes from its ability to ignore specifics in favor of reproducing general patterns.
To be useful tools for writing papers, running businesses, or targeting ballistic missiles, LLM outputs MUST be accurate. But to make every output accurate, we need to have giant models that require even more compute and electricity. Perfect model output would also require multiple models from different vendors working simultaneously to check each other. When AI is already the most expensive science experiment that humanity has ever funded, there is not a viable path to give AI control of critical real-world systems with our current model architecture and GPU data centers.
If we do find a way to make an AI model that performs perfectly at a viable cost/scale, this will open up another set of even bigger problems. A perfect LLM will be able to perfectly optimize itself as our friends on the HBO show Silicon Valley discovered in the series finale.
It’s not all bad news. There are tasks AI can do very well that don’t need to be perfect. A 95% accuracy rate for a translation app that lets you point your phone at a menu in a foreign language is unlikely to hurt anyone. Worst case, you might get boiled chicken instead of delicious fried chicken. If a local newspaper can’t afford human writers, a 95% accurate AI-written story that is generated from the audio recording of a city hall meetings is better than no news coverage of the meeting.
Meanwhile, a 95% accuracy rate is a catastrophe for a commercial flight path, threat assessment, or a military targeting system.
If AI is going to be used to run the US military, diagnose medical conditions, map flight routes, drive cars, manage supply chains, or the millions of other business functions that are being farmed out to Google, OpenAI, and Anthropic, we either need to accept that major errors will happen…or we need to slow down the AI integration across our society.
The motto in Silicon Valley has been “move fast and break things”. I don’t want to see another 170 graves being dug next to a school so we need to slow down and do it right.
David Riedman, PhD is the creator of the K-12 School Shooting Database, Chief Data Officer at a global risk management firm, and a tenure-track professor. Listen to my podcast—Riedman Report: Risk, AI, Education & Security—or my interviews on Freakonomics Radio and the New England Journal of Medicine.