Meta's randomized review-UI trial measured reviewer eyeball time—and changed who saw the patch

Original and work dates. Chandra Maddila et al., AI-Assisted Fixes to Code Review Comments at Scale, posted July 17, 2025. The two Meta Phabricator trials ran October 17–November 21, 2024 and November 22–December 2024 (Table I ends the second December 20, prose says December 18). They concern AI patches offered in response to human reviewer comments, not AI-authored pull requests nor autonomous detection of missing requirements. The paper does not report arm sample sizes, mean minutes or a post-release defect comparison; internal change-level data and benchmarks are not public.

Why this is different from a PR clock. The team monitored three safety metrics: TimeInReview (time waiting for and in review, excluding author rework), TimeSpent (time reviewers spent viewing the Phabricator review page, their ‘eyeball time’ proxy), and WallClock (submit until accepted or abandoned, including author fixes). The first randomized safety trial compared no AI patch with a GPT-4o-generated suggestion visible to reviewer and author. Table IV gives +6.7% TimeSpent (p<.001), +5.5% TimeInReview (p=.029), +3.0% WallClock (p=.095). Text mistakenly repeats the 5.5% and p=.029 numbers as if they also describe reviewer eyeball time; use the table and flag the discrepancy. Engineer feedback indicated reviewers felt obliged to vet the optional generated patch, including when it was only viewed rather than applied. That is a plausible mechanism from interviews and dose association, not separately randomized proof of individual cognition.

Second intervention, not a time-saving claim. The redesign kept author-facing patches but collapsed the suggestions in reviewers’ view, with an option to expand. A second no-AI-versus-redesigned-UX randomized trial found +1.24% TimeSpent (p=.66), +0.42% TimeInReview (p=.86), +0.26% WallClock (p=.67) in Table IV. Failure to detect a difference is not proof that time was identical; no confidence intervals or minute baselines are given. The two trial phases differ in calendar time and presentation, so the study demonstrates two within-phase contrasts against no AI, not a direct randomized comparison of reviewer-visible versus author-focused designs. Across those phases about one-tenth of machine-classified actionable comments had applied AI patches; denominator is actionability classification, not all requests or user issues. Later all-engineer rollouts switched models and were not randomized; the stronger model's 19.75% actionable→applied is not a customer-confirmed success rate.

What counts as preserved warning? Generated patches had to pass build/lint/tests or receive explicit reviewer endorsement to be shown; author can accept through review UI, IDE or copy/paste, and detection of latter is heuristic. Thus applied ≠ independently checked behavior. Suggestions could be archived as applied, discarded or made irrelevant by a changed diff; recording the reason helps trace individual patch disposition but does not assign an owner to a human-accepted unfixed concern or follow it to release. There is no reported author-active-time effect after the interface change, review correctness, task cost or maintenance outcome. Page viewing also misses off-screen reading, testing and conversations; a reduction in UI eyeball time can move work out of sight.

Relevance. This is the strongest primary partial answer to active review effort: a real field randomized study detects extra reviewer-page attention from the visibility design, while Chen–Stratton measures the different submit→merge queue after firm coding-agent adoption. Neither identifies whether the same assisted request produced independently accepted behavior, and shifting the patch to the author might shift responsibility or labor rather than save it. Compare a reviewer-visible and an author-focused arm prospectively on paired reviewer/author active minutes, detection of seeded and naturally discovered issues, release acceptance and later repair.

Feed: original paper is a weekend long read after the October 10 Chen–Stratton paper is opened/hearted or the review question returns. For a weekday, a short original account of the UI choice could work if it carries the active-minute distinction honestly; avoid a vendor announcement.