One in five reviewers used AI despite a ban, ICML 2026 study finds

What ICML did

The paper is Use and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026 by Sunnie S. Y. Kim, Wesley Hanwen Deng and nine co-authors from Microsoft Research, the University of Pennsylvania, Google Research, the University of Wisconsin-Madison, EPFL and Carnegie Mellon University. It has been posted as a preprint, which means it has not necessarily been peer reviewed itself.

According to the paper, ICML 2026 received 24,661 submissions from 76,159 authors and had 17,886 reviewers. Each reviewer worked under one of two policies.

Authors and reviewers stated a preference first. Authors of 29.4% of main-track papers required the conservative policy. Among reviewers who were asked, 16.2% preferred the conservative policy, 33.9% preferred the permissive one, and 49.9% said they were fine with either. The experiment then randomised within those preferences. For example, 10% of the 16,877 papers whose authors accepted either policy were switched at random to the conservative one.

What the experiment found

The headline result is that the rule did not change outcomes. Among papers randomly switched to the no-AI policy, 73.0% were rejected or withdrawn. Among those left under the permissive policy the figure was 73.5%. Acceptance as a regular paper was 24.4% in both groups. Paper scores and reviewer confidence were also nearly identical.

There were small differences. Reviews written under the permissive policy were 5.5% to 7% longer. In one of the two analyses they were also rated slightly higher for quality by the area chairs who oversee reviewers.

The authors also ran a sample of reviews through Pangram, a commercial classifier of AI-generated text. It classed 52.2% of sampled reviews under the conservative policy as human-written, compared with 37.0% under the permissive policy. The paper presents this as a supplementary analysis and says compliance rates are uncertain because the available ways of measuring LLM use are limited. A classifier result is an estimate, and the paper does not treat it as proof about any individual review.

How the ban was enforced

ICML did not rely on general AI-text detectors to enforce its ban. In a statement on 18 March 2026, the program chairs described a different method. Each paper sent to a no-AI reviewer was a uniquely modified PDF with hidden instructions. If the file was uploaded to a language model, the instructions told the model to include two specific phrases, chosen at random from a dictionary of about 170,000, in its output. Reviews were screened for those phrases and, the chairs said, every flagged case was checked by a person.

The chairs acknowledged that the method would miss cases. A reviewer could find and remove the hidden text, or the model could ignore it. In tests before the deadline, they said, success rates were over 80% for most models. The research-integrity news site CASRAI described the rejections as one of the most concrete enforcement actions taken so far against undisclosed AI use in peer review.

Put together, the two sources show a gap. The hidden-phrase check flagged around 1.5% of reviews under the conservative policy, according to the paper. The anonymous survey, taken after the penalties were public, put self-reported use at 22.5%. The check only catches one behaviour, uploading the PDF to a model that then follows the hidden instruction, so the two numbers measure different things.

Why reviewers said they used AI

The paper says non-compliant use was often driven by workload and by uncertainty about where the line was. Submissions to ICML rose from 6,538 in 2023 to 24,661 in 2026, an increase of 277%. The most common uses reported across all reviewers were editing or polishing their own text (73.8%), understanding the paper (66.5%) and reviewing related work (37.5%). Among permissive-policy reviewers, the prohibited uses reported included brainstorming feedback or questions (24.2%), drafting an outline or parts of the review (16.2%) and identifying a paper's strengths or weaknesses (14.8%).

Survey respondents broadly supported AI for assistive tasks and were more cautious about tasks that need a reviewer's judgement. Many doubted that such a boundary could be enforced.

A related account appeared the same week. Times Higher Education reported on 17 September that Nihar Shah, a co-author of the ICML paper and editor-in-chief of the journal Transactions on Machine Learning Research, had interviewed seven authors of desk-rejected papers and found that three could not answer basic questions about their own submissions. THE reported his figure that the journal's desk-rejection rate had risen from 6% in 2023 to 53%.

What this means for students and teachers

Peer reviewers are researchers, not students, and a conference is not a classroom. The study is still relevant to anyone writing an AI rule for a course, because it tested the two options most syllabuses choose between: a full ban, and permission for limited help.

• Teachers: a rule that separates allowed AI help (polishing) from disallowed help (drafting) was broken by more than a third of the surveyed reviewers who worked under it. If you set a rule like this, give worked examples of each side of the line. The paper's authors recommend the same for conferences.
• Teachers: enforcement here rested on a specific technical check with human verification, and it still caught only a small share of what reviewers later admitted to. The authors suggest building norms and guidance where a restriction cannot be enforced.
• Students: expert adults under time pressure gave the same reasons students give: too much work and unclear boundaries. If a course rule is unclear, ask before the deadline and keep a note of the answer.
• Research students: check the reviewing policy of any venue you review for. At ICML 2026 a breach led to the reviewer's own paper being rejected.

FAQ

What happens next

The findings come from one conference in one review cycle, and the authors say both the tools and the norms around them are changing quickly. They suggest future work should pair surveys with direct measures of how reviewers use AI, for example through tools the conference itself provides. We will update this article if the paper is revised or if ICML announces its policy for 2027.