The verification bottleneck AI forgot to remove

Maxim Topaz used an AI chatbot to help edit an editorial he was submitting to a journal. He checked the citations himself, carefully, the way anyone would after hearing enough horror stories about AI-invented references. An editor at the journal still flagged one that did not exist. Topaz, a nurse and health AI researcher at Columbia, was embarrassed enough by the near-miss that he went looking for how common the problem actually was. He sifted through more than two million papers and ninety-seven million citations and found the rate of fabricated references had risen sixfold between 2023 and 2025, then kept climbing: by early 2026, roughly one in every 277 papers contained a citation to a source that does not exist.

A few months later and an ocean away, a medical statistician named Adrian Barnett was running reverse image searches on a Kaggle dataset called "Facial droop and facial paralysis image," used by multiple teams to train stroke-detection models. He matched several of the photos to a dental journal article from years earlier. The images were not of stroke patients. Some were of people with an unrelated condition, Bell's palsy. Some were stock photos. Some were celebrities. One showed a minor whose guardians had consented to a clinical publication, not to becoming open training data for anyone who wanted to boost their standing on a machine-learning leaderboard. Barnett flagged this to Kaggle in February. Nothing happened. It was only when the journal itself complained about copyright infringement, five months later, that the dataset came down. "I think it's funny," Barnett said afterward, "because we approached Kaggle and said, 'Hey, this dataset is rubbish, scientific nonsense,' and that didn't work."

These are not the same story, but they are the same shape. Somewhere between the moment a piece of research work is produced and the moment it becomes part of the permanent record, a check used to happen. It happened almost by accident: writing a citation by hand meant locating the source, because there was no faster way to get the details right. Assembling a training set meant looking at the images, one by one, because there was no way to gather a thousand of them without opening a thousand folders. The verification was not a separate step anyone budgeted for. It was baked into how slow the work was.

AI removed that friction on the production side. It did not add anything equivalent on the verification side. A citation can now be generated as easily as a plausible-sounding one, which is not the same thing as a real one, and nothing about the tool that generated it forces a check before it goes in a manuscript. A dataset can now be scraped and packaged at a scale no individual would attempt by hand, and nothing about the platform hosting it verifies that the images are what the folder name claims. The people involved in both stories were not careless by the standards of ten years ago. They were operating a system that quietly stopped including verification as a built-in cost of production, and almost nobody has yet decided whose job it is to add that cost back.

The obvious answer is that automated verification will simply scale up to match, the way spam filters eventually caught up with spam. There is some evidence for this: several major biomedical publishers now run automated reference-checking tools before a manuscript reaches print, and journal indexers have begun requiring evidence of citation-screening as a condition of continued listing. But the publishers actually piloting these tools describe a more complicated picture than an arms race quietly won. The Public Library of Science reported seeing large numbers of false positives, legitimate references flagged because of formatting quirks or incomplete database coverage, which is its own kind of cost. And even a flawless citation-checker would have done nothing for the Kaggle dataset, because that failure was never about the mechanics of a reference list. It was about whether a folder of images was what it claimed to be, a question that sits one layer beneath anything a citation tool can see. Two failures with the same underlying shape do not have the same fix, which is itself a sign that the shared cause is not a missing piece of software but a missing decision about where responsibility now lives.

Rigour was never a personal virtue that researchers of an earlier generation simply possessed in greater supply. It was mostly a byproduct of how much friction stood between wanting to publish something and actually being able to. Remove the friction and the rigour that leaned on it does not degrade gracefully. It disappears, quietly, until someone with Barnett's kind of stubbornness goes looking for it by hand. Rebuilding it will mean treating verification as a designed feature with an owner and a budget, not a habit that is supposed to reassert itself once enough people feel embarrassed enough times. Nobody has assigned that ownership yet. Until someone does, the next version of this story is not a matter of if.