ChatGPT's Limits and Hallucinations

What ChatGPT Can’t Do, and Why It Matters

Knowing how to use ChatGPT starts with knowing where it fails. The tool is powerful, but it has flaws that experience alone won’t fix—you have to work around them. This lesson is arguably the most critical in the course because it separates users who verify information from those who will one day confidently spread misinformation. OpenAI states this clearly in its Terms of Use: a response may be inaccurate, it should not be your sole source of truth or replace professional advice, and you are responsible for evaluating its accuracy.

Hallucinations: When the Model Makes Things Up

A hallucination is a response that looks perfectly correct—confident phrasing and flawless structure—but is factually wrong. ChatGPT isn’t lying; it generates the most probable continuation based on learned patterns, and that continuation is sometimes incorrect. When it doesn’t “know” something, it doesn’t always stop to say so; it may generate a plausible-sounding answer instead.

OpenAI describes the most common forms: incorrect definitions, dates, or facts; made-up quotes, studies, or references attributed to non-existent sources; and overconfident answers to ambiguous or complex questions. There’s another trap many discover the hard way: ChatGPT cannot reliably tell whether a text was written by an AI, or whether it wrote the text itself. Its answer to “Did you write this essay?” proves nothing.

The defense comes down to four habits. Independently verify any critical information: numbers, dates, names, sources. Let the right tools ground the response: use web search for current events and specific facts, data analysis for calculations, and deep research to cross-check multiple sources. Ask for sources, then open them: a link that doesn’t support the claim is as good as no source at all. Finally, raise the level of reasoning on tricky questions: you won’t eliminate the risk, but you give the model time to walk through its logic.

What the Model Cannot Know

The model’s knowledge stops at a cutoff date. Without web search, anything that happened after that date is invisible to it, and it may not even realize it’s missing information. With web search enabled, another limit appears: some sites remain inaccessible due to technical issues, paywalls, or because they block bots.

Coverage also varies by topic. On well-documented subjects, answers are often excellent; on niche or rarely written-about topics, quality can drop sharply, and nothing in the tone will warn you. OpenAI also flags biases: presenting a single perspective as absolute truth, oversimplifying nuanced issues, or misrepresenting the weight of scientific consensus or societal debate. On sensitive topics, cross-check with other sources.

Reasoning Limits

Multi-step calculations remain a weak spot when done “in your head” within the response text. For any calculation that affects a decision, ask the model to use data analysis, which executes code, and cross-check with a dedicated tool. Very long reasoning chains carry the same risk: the thread gets lost, and inconsistencies creep in along the way. Counting is another counterintuitive flaw: tallying letters, words, or items can trip up the model, even though a child could do it.

Practical Limits of a Conversation

Every conversation has a context window—the amount of information ChatGPT can use in that thread. Your messages and its responses both take up space, and a few very long exchanges can consume more than many short ones: so there’s no fixed number of messages per conversation. The size of this window increases with your plan. When it’s full, ChatGPT tells you (“Context window full”): you can still review the thread, then immediately start a new conversation, optionally with a summary provided by ChatGPT. That summary might drop an old instruction or exact wording: reattach what matters.

The same effect applies to long outputs: if you have the model write a twenty-page document section by section, contradictions can appear between parts. A full review isn’t optional, and flagged inconsistencies can be fixed in a single message.

Try It Yourself

Test the limits yourself. Ask for a specific quote from a French author, with its source, then look it up yourself using a search engine. Ask about a very recent event without web search, then ask again with search enabled. Have the model do a five-step calculation “in its head,” then via data analysis, and redo it by hand. Finally, ask how many times the letter R appears in “extraordinary,” then count yourself. These exercises build an intuitive sense—more reliable than any memorized rule—of when you need to verify.

Two pitfalls await. First, blind trust: the higher the stakes, the tighter your controls must be. Second, the opposite extreme: blaming the tool when your prompt was vague. Before concluding there’s an error, rephrase your request.

Official sources: Does ChatGPT tell the truth? (OpenAI Help Center: hallucinations, tools, cutoff date, biases, AI text detection) · ChatGPT Free Tier FAQ (full conversation) · Europe Terms of Use (response accuracy) · Data analysis with ChatGPT (calculations executed in Python)

Key Takeaways

• Hallucinations stem from how ChatGPT works: it generates plausible text, not necessarily true text
• Always verify critical facts, quotes, sources, and numbers; ChatGPT cannot reliably detect AI-written text
• Web search, data analysis, and deep research reduce risk but don’t eliminate it
• Conversations have a limited context window: when it’s full, start a new thread with the essentials
• Knowing these limits doesn’t reduce ChatGPT’s usefulness—it makes you more effective