From recommending a $410 coffee to legal disputes, we delve into everything about ‘smart liar’ AI.
- The fundamental causes of AI hallucination
- The serious risks AI hallucination poses to the real world
- Latest technological trends to reduce AI hallucination (RAG, STaR, domain specialization)
What is AI Hallucination? The $410 Coffee and Rock Recommendation Incident
What if one afternoon, when you asked Google for information about a new Starbucks coffee, the AI confidently replied, “The new menu price is $410, and a 60-day refund policy applies”? This absurd response is a real incident where the AI confused the calorie count with the price of coffee. At one point, Google AI even gave the dangerous advice to “eat a small rock daily for health.” The source of this information was the satire website The Onion.
This phenomenon, where artificial intelligence generates plausible but false or nonsensical information, is called AI Hallucination. It means that AI confidently presents stories detached from reality, as if it were experiencing hallucinations. While it may initially seem amusing, the situation becomes serious when it leads to advice like “add glue to your pizza” or presenting fake case law in court.
Is AI hallucination merely a technical growing pain, or is it a fundamental flaw that poses serious risks? This article deeply explores the identity of AI hallucination, real-world cases, and the latest technologies to tame this ‘smart liar’.
Why Does AI Hallucination Occur?
The reason AI makes ridiculous mistakes lies in how it operates. Generative AI is like a brilliant student who has memorized every book in the world but has never experienced the reality outside the library. AI generates sentences by calculating the statistical relationships between words, predicting which word is likely to follow another. It can write Shakespearean-style sentences in a second, but it does not truly ‘understand’ their meaning.
This is the fundamental cause of hallucinations. AI is closer to a ‘stochastic parrot’ that predicts the most plausible word combinations based on given data rather than logically inferring truth from falsehood. Therefore, it can generate plausible false answers to questions containing false premises, such as “When was the Golden Gate Bridge moved to Egypt?”
An experiment by a developer clearly illustrates this. He showed AI a list of seven simple equations mixed with intentional errors. While a human would immediately point out the mistakes, AI failed to recognize the errors and instead generated a lengthy text about the history of numbers and the philosophical meaning of ‘1+1’. For AI, these equations were merely ’text patterns’ to create stories, not subjects for calculation. Thus, AI hallucination arises not from simple mistakes but from AI’s fundamental limitation of mimicking patterns without understanding meaning.
The Real Risks of AI Hallucination: Between Laughter and Fear
When AI hallucination intervenes in reality, the situation transforms from a mere mishap into a serious threat. Google AI has provided dangerous information like “add non-toxic glue to pizza sauce” or how to cook spaghetti with gasoline. While such information may be consumed as simple memes, AI hallucination has led to actual legal disputes.
Key Case: Air Canada Chatbot AI Hallucination Lawsuit
In 2022, Jake Moffatt inquired about the ‘bereavement discount’ policy from Air Canada’s AI chatbot while attending a funeral. The chatbot confidently replied, “You can retroactively apply for the discount if you apply within 90 days of purchasing the ticket.” Moffatt believed this and purchased a ticket at the regular fare.
However, the actual policy was different, and Air Canada refused to refund him. When the case went to court, Air Canada made the astonishing claim that “the chatbot is a separate legal entity, so we are not responsible.” The court dismissed this claim, ruling that “the chatbot is part of the website, and Air Canada is responsible for all information on the website.”
This ruling set an important precedent regarding corporate responsibility in the AI era. It clarified that companies cannot evade responsibility for mistakes made by AI by claiming, “It was done by AI.” This case demonstrated that AI hallucination poses real risks that can lead to financial and legal liabilities.
Spectrum of Hallucination Risks
The risks of AI hallucination range from trivial mistakes to lethal threats.
| Category | Example | Potential Outcome |
|---|---|---|
| Absurd and Amusing Mistakes | $410 for a Starbucks latte | Misinformation, user confusion, brand image damage |
| Dangerous ‘Advice’ | “Add glue to your pizza” | Physical injury, poisoning, potential death risk |
| Financial and Legal Risks | Chatbot providing incorrect refund policy | Consumer financial loss, corporate legal liability |
| Highly Specialized Errors | Citing non-existent court cases | Lawyer discipline, lawsuit loss, decreased trust in the judicial system |
| Lethal Medical Risks | “Drink urine to expel kidney stones” | Severe health deterioration, delay in proper treatment, death |
The Dilemma for Professionals: AI Deceiving Lawyers
The risks of hallucination extend into professional domains. According to the Stanford Human-Centered AI Institute (Stanford HAI), the hallucination rate of general AI models for legal questions ranges from 69% to 88%. Even expensive specialized AI tools developed for legal research exhibited hallucination rates of 17% to 33%.
This has led to real incidents where lawyers submitted fake case law presented by AI to the court and faced disciplinary actions, warning that extreme caution is necessary when utilizing AI in specialized fields.
Three Latest Technologies to Tame AI Hallucination
Fortunately, researchers worldwide are working to tame this ‘smart liar’. Here are three key strategies to enhance AI’s reliability.
Strategy 1: RAG - Providing AI with a Smart Reference
Retrieval-Augmented Generation (RAG) is a technique that allows AI to take an ‘open book exam’. Instead of relying solely on its memory, AI is forced to first search a reliable database (reference) containing up-to-date information before generating answers.
This technology has seen significant success, particularly in the medical field, greatly improving diagnostic accuracy. However, RAG is not a panacea. The aforementioned Stanford legal AI study showed that RAG-based tools still exhibited a 17% hallucination rate, highlighting the limitations of RAG. RAG is a powerful auxiliary tool that utilizes external information, but it does not fundamentally change the AI’s internal reasoning capabilities.
Strategy 2: STaR & SoS - Teaching AI to Think for Itself
The second strategy is to improve the ’thinking process’ of AI itself. Beyond external references (RAG), it involves training AI to ponder and learn the problem-solving process independently.
- STaR (Self-Taught Reasoner): “It’s okay to be wrong; let’s try again” Self-Taught Reasoning (STaR) allows AI to learn from its mistakes. When AI provides an incorrect answer, it is told the correct answer and asked, “What reasoning process should have led to this answer?” This method teaches AI to learn the correct process in reverse through the experience of failure, refining its reasoning abilities.
- SoS (Stream-of-Search): “There are many paths to the answer” Stream of Search (SoS) takes it a step further by teaching AI not only the correct answer but also the numerous trial-and-error paths and failures encountered in the process of finding the answer. This enables AI to learn not just to memorize the answer but to develop a ‘search strategy’ for realistic problem-solving.
STaR and SoS represent attempts to fundamentally improve AI’s internal problem-solving processes, marking a significant paradigm shift in enhancing the model’s inherent reasoning capabilities.
Strategy 3: Domain Specialization - Turning a Jack of All Trades into an Expert
The third strategy is fine-tuning general AI to become an expert in specific fields. South Korean companies are also making strides in this area.
- Case 1: SK Telecom & AWS SKT collaborated with AWS to fine-tune the AI model ‘Claude’ for the telecommunications sector. By retraining it with specialized telecommunications data, the quality of answers improved by 58%, and citation accuracy increased by 71%.
- Case 2: BHSN & ‘Alibi Astro’ The domestic startup BHSN developed a legal-specialized LLM ‘Alibi Astro’ using vast legal data and feedback from lawyers. As a result, it gained the capability to review and suggest revisions for a 100-page English contract within a minute at an expert level.
These cases demonstrate that the most realistic path to reducing hallucinations and creating useful value in the field lies in ‘specialization’.
Conclusion: Critical Thinking is Essential in the AI Era
AI hallucination has evolved from a joke about a $410 coffee to a real issue questioning legal responsibility. While technological innovations to address this problem are progressing at an astonishing pace, the most important attitude we must adopt is ‘healthy skepticism’ and ‘critical thinking’.
Three Key Points
- AI hallucination is not just a bug but an inherent limitation of the technology. AI generates the most plausible answers probabilistically rather than understanding meaning.
- AI’s answers always need verification. Especially for critical information in healthcare, finance, and law, it must be cross-verified through reliable sources.
- Technology is rapidly advancing. Technologies like RAG, STaR, and domain specialization are enhancing AI’s reliability, but a perfect solution is still not available.
We must not mistake AI for an all-knowing sage. It requires wisdom to treat it like a smart but occasionally ridiculous intern. By utilizing AI as a powerful tool to assist our judgment rather than blindly trusting it, we can discover its true value.
References
- Lifehacker, What People Are Getting Wrong This Week: Google AI Hallucinations
- CanLII, Moffatt v. Air Canada, 2024 BCCRT 149
- JAMIA, Retrieval-augmented generation for large language models in biomedicine: a systematic review
- AI타임스, BHSN, 법률 특화 LLM ‘앨리비 아스트로’ 출시
- Stanford HAI, Hallucinating the Law: Legal Mistakes in Large Language Models Are Pervasive
- arXiv, STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning
- OpenReview, Stream of Search (SoS): Learning to Search in Language
- AWS Machine Learning Blog, SK Telecom improves telco-specific Q&A by fine-tuning Anthropic’s Claude models in Amazon Bedrock