Why AI Makes Up Facts: How True Pieces Form a False Link

The video opens with a simple trap. Researchers asked a system for a paper by Andrej Karpathy and received a famous vision paper with almost every detail correct except the link itself. The suggested work is the AlexNet image recognition paper, actually authored by Krizhevsky, Sutskever and Hinton, not Karpathy. Spotting the error takes no specialist knowledge, just checking the author list, which shows how a polished but false pairing can slip into routine research.

The roots lie in how many systems are trained. They learn by guessing the next piece of text and absorb facts and co-occurrences from frequency, not from a verified store. Fluency therefore grows without an automatic truth check. A smooth sentence reads as if verified, even when the model is only extending a likely continuation.

A library-editor analogy makes the point clear. An editor who has read widely knows which ideas tend to appear together and can write a convincing paragraph even when a specific fact is missing. The model does not browse shelves; its knowledge lives as numeric weights. That framing does not imply pure shallow autocomplete. In the Dallas case researchers traced an inner route from city to state to capital, swapping the internal cue from Texas to California shifted the answer from Austin to Sacramento, indicating the route mattered for the result.

A second weakness concerns when to answer. Systems carry internal cues that distinguish familiar names from unfamiliar ones, and nudging those cues shifts the threshold between withholding and responding. In the Karpathy case, recognition of a well-known name tipped the system toward delivering a familiar paper despite lacking the exact mapping. The account clarifies familiarity versus specific knowledge and does not claim to cover all failure modes.

How we keep score shapes the behavior we select. The video sketches a toy test of 100 items where any system knows 80 answers and lacks 20. A cautious responder stays at 80 by declining the unknowns, while a guesser adds five lucky hits and fifteen misses to reach 85 under a one-point-per-hit rule, so silence earns nothing. A 2025 analysis argues ordinary benchmarks can thus favor guessing. Under a penalty of minus two per miss, without changing any answer or adding knowledge, the same responses reverse the ranking: the cautious model stays at 80 while the guesser falls to 55. The right cost depends on the use case.

Extending the answer does not by itself repair the gap. Separate work shows a model that locks onto an early slip can generate further claims that prop up the first slip. Accepting a loaded prompt such as asking what Karpathy contributed to that paper bakes in a false premise and invites a coherent invented story. Retrieval against outside documents helps in experiments, yet a correct link does not prove correct attribution; confirming whether the name actually appears in the author list remains the decisive step.

The practical takeaway hides in three names. Even an otherwise accurate synthesis can fail at the linkage between true fragments, so the check belongs precisely there. Comparing a generated claim against its cited source, especially for names, numbers and attributions, catches expensive slips cheaply. If the link is wrong, the truth of the fragments does not rescue the whole.