AI models caught up to 99% of major errors planted in clinical-trial reports
A medRxiv preprint says 15 LLMs beat a 27% human-review baseline, though the study used LLM judges and also found substantial control-report flags.
Fifteen large language models detected between 58% and 99% of planted major methodological errors in reports from 58 randomized clinical trials, according to a new medRxiv preprint. In the earlier human experiment that the benchmark recreated, 607 untrained reviewers found an average of 27% of the major errors.
The researchers inserted known errors into recent trial reports, then generated 1,796 reviews. Even the four open-weight models scored between 66% and 89%, while review times across all models ranged from 1.4 to 16.3 minutes. The result suggests AI can serve as a fast second reader for clinical research and may catch problems that conventional peer review misses.
The finding is not evidence that AI should replace reviewers. The paper has not been peer reviewed, and two blinded LLM judges scored the generated reviews. Models also flagged 14% to 29% of the tested error types in unmodified control reports; some may be genuine issues, but the study cannot treat every flag as a confirmed error. Four authors are affiliated with ResearchHub Foundation, which is implementing AI-assisted review. The practical takeaway is narrower: human-only review may leave avoidable mistakes behind, while AI-assisted review deserves controlled use and independent validation.
