AI hallucinations in code, and how to catch them

A hallucination is a model producing something that fits the shape of a correct answer without being one. In code it is rarely gibberish. It is a package that sounds exactly like a real package, a method that ought to exist on a library, or a config option that would make perfect sense if anyone had implemented it.

That plausibility is the problem. Nonsense gets caught immediately. A confident, well-named, wrong answer gets committed.

The check everyone recommends does not work

The standard advice is to verify that a suggested package exists before installing it. So here is that check, run against a set of plausible-sounding names on the npm registry:

Every name tested was already registered. Including lodash-utils with five weekly downloads and string-utils-pro with seven, which are names invented for this test.

The npm namespace is saturated. Any plausible combination of words already resolves to something, usually an abandoned package from years ago with single-digit usage. So "does it exist" answers nothing, and npm install succeeding is not evidence you got the right package. It is barely evidence you got a real one.

The signal is not existence. It is usage: downloads, publish date, repository, and whether anyone depends on it. use-debounce at 6.4 million weekly downloads and react-use-debounce at 63 are not two options. One is the library and one is noise sitting next to it in the namespace.

Slopsquatting

This gets worse than accidental. Attackers watch which package names models tend to invent, register those names, and publish something malicious under them. The name is known as slopsquatting, by analogy with typosquatting, and it works because the model reliably sends people to a name that did not previously exist.

The attack needs nothing clever. It only needs the model to be predictable, which it is, and for someone to install without looking, which is the default behaviour.

That is why treating a hallucinated package as merely an annoyance is the wrong instinct. An E404 from an npm install is the safe outcome, because it means nobody has claimed the name yet. The install that succeeds quietly is the one to think about.

Three checks, in order of speed:

1. Look at the weekly download count on npmjs.com. Anything in single or double digits for a library that supposedly solves a common problem is wrong.
2. Open the repository link. No repository, or a repository with no commits in years and no issues, is a strong signal.
3. Check who depends on it. A real utility library has dependents. An invented one has none.

None of that takes longer than a minute, and it is a minute you spend once per new dependency, not once per suggestion.

Hallucinated methods on real libraries

The subtler version. The library is genuine, the import works, and the method does not exist:

import { format } from "date-fns";
format(new Date(), "yyyy-MM-dd");   // real

array.groupBy((x) => x.type);        // not a real Array method

These surface as is not a function at runtime, which is a good outcome because it fails loudly. The bad outcome is when the method exists but does something different from what the model described, in which case you get plausible wrong behaviour and no error at all.

The reliable defence here is the documentation, not another prompt. Asking a model whether a method exists has the same failure mode as asking it for the method in the first place: it will produce a confident answer either way, and confidence is not correlated with correctness.

Version drift

The most common hallucination is not invented at all. It is correct code for a version you are not using.

A model's sense of a library is an average of everything it read, weighted toward whatever was most written about. For a library that changed its API, that average is often the old one, so you get code that was genuinely correct in v4 presented as current for v6.

This is why version-specific questions have the worst hit rate, and why the answer is always the changelog. It is also the reason to state your versions in the prompt, which helps, but does not make the answer trustworthy on its own.

The checks that actually catch these

Ranked by how much they catch per unit of effort:

Run the code. Most hallucinated APIs fail immediately. This is the cheapest and most effective filter, and it is skipped surprisingly often for code that looks obviously right.

TypeScript. A hallucinated method on a typed library is a compile error before you run anything. This is the single biggest structural defence, and it turns a class of runtime surprise into an editor squiggle.

A linter with import resolution. A rule such as import/no-unresolved flags an import that does not exist, catching the package case before install rather than after.

Lockfiles and review. Every new dependency should be visible in a diff, and a package-lock.json change is exactly where a slopsquatted package becomes reviewable. A dependency added without anyone reading the name is the failure this whole page is about.

Tests for the behaviour, not the shape. Code that runs is not code that is correct. The version-drift case in particular produces code that executes happily and does the wrong thing.

What a hallucination looks like in review

The reason these get merged is that they do not look like mistakes. They look like the work of someone slightly more experienced than you, which makes them harder to question rather than easier.

Four signatures worth recognising in a diff:

A new dependency you have not heard of, solving a problem you thought was easy. Real ecosystems have one or two obvious libraries per job. A third name appearing for a common task is worth thirty seconds on npmjs.com.

An API that is suspiciously convenient. A single method that does exactly the multi-step thing you needed, with exactly the argument order you would have chosen, is a plausible thing for a model to construct and a rare thing for a library to ship.

Config options that read like documentation. { retryOnFailure: true, backoff: "exponential" } is what the option should be called. Whether it is depends on the library, and this is the case that fails silently, because unknown options are usually ignored rather than rejected.

Confident comments explaining incorrect code. The explanation is generated alongside the code and inherits the same error, so a wrong line often arrives with a fluent comment justifying it. A comment is not evidence.

The common thread is fluency. In human code, the confident and specific parts are usually the parts the author knew best. In generated code that correlation does not hold, and the most authoritative-sounding details are the ones most worth checking.

The habit underneath all of it

Hallucinations are not a bug that will be patched away. They are a consequence of how these models work: they produce likely text, and a plausible package name is likely text whether or not it exists.

So the working assumption is that generated code is a draft by a confident stranger. It is often good. It is not verified, and the parts that look most authoritative, such as a precise-sounding package name or a specific config option, are exactly the parts most worth checking, because that is where fluency does the most damage.

The practical version fits in a sentence: run it, type it, and look up anything you are about to install. If you want the broader version of that argument, vibe coding and where it breaks covers what happens when nobody does.

Want to build the judgement that makes this quick? Start with a track and write enough of it yourself to recognise when something is off.