When a skill rule changes: required action, discovery and delivered correctness

Research paper read October 23 for weekend consideration. Jiajie Wang and six colleagues, “Agent Skill Evolution: How Revisions Affect Coding Agents”, arXiv version October 4, 2026. This is a controlled comparison of versions of public project skill files, not a longitudinal deployment or an evaluation of October 22 foundation governance. It tests the missing middle of portable skills versus durable decisions: whether an agent retrieves and acts on a newly added rule.

Sampling and design. Authors rebuilt revision histories for 3,159 SKILL.md files (856 repositories); 2,608 first/last pairs differ. A subset of 396 probes came from lines that added a mechanically checkable command/path/identifier; an LLM generated requests where that rule would apply, with the required string absent from the request. Old and new versions faced the same request and model. Their 80-task mini-SWE-agent sandbox, drawn from 57 repositories, held the code at the newer commit, with three runs per condition for open models and one for Claude Sonnet 4.5; 36 tasks supported a separate automated final-artifact check. Paired setup isolates changed instruction text conditional on a selected request needing the rule, not selection of skills or frequency of those requests in ordinary use. A stratified human sample checked the literal-string grader; separate blind adjudicators judged final artifacts.

Findings in the sandbox. For four agent settings, explicitly supplying the revised skill body raised the required-action rate by 0.16 to 0.36 (mean +0.23), with Sonnet 4.5 old 0.62 → new 0.81. On-demand discovery reduced the four-agent mean gain to +0.12, about half the preloaded effect; Sonnet preserved +0.19 but three open models retained about 38% of their preloaded gain. Qwen 3.5-27B and Sonnet opened the revised skill on 89% and 88% of episodes, respectively; the relevant added line actually appeared on screen in about 71% and 69% of reads, in part because the sandbox clipped long command outputs. Conditional on a displayed rule, required action was 61% for Qwen and 88% for Sonnet. Read ≠ seen ≠ acted on ≠ accepted outcome. Blind judges found final artifact correctness across all 80 tasks improved by +0.14 and +0.11 for the two Qwen models; the Sonnet change was +0.06 with an interval crossing zero. The separate automated correctness check on 36 tasks found Sonnet +0.03 with an interval crossing zero. Preloading a skill body increased token use per agent episode by 37–57%; a revision beyond the old body added no detectable episode tokens in this setup. That is not an invoice for maintaining skill quality or human review.

Limitations and disagreement. A literal token can be echoed without correct action; the sandbox action measure checks whether a command/file is used, not whether the eventual software helps users. Blind outcome adjudication supports two open models more than the 36-task automated artifact check, which only moderately agreed with human labels. The mechanically selected directives underrepresent prohibitions, conditional permissions and subtle semantic intent; the revised files often differ in several places and the dataset is more popular than the registry overall. The benchmark’s networkless environment has no release, on-call operator, incident reporter or wrong-permission risk. Kiro’s first-party production account reports per-cell playbook correction and bad corrections contaminating later sessions, but does not provide a public same-alarm pre/post log to quantify transfer. The paper supplies a causal-in-sandbox reason to write discoverable, action-specific lessons, while Kiro supplies a reason to validate what gets written. They do not prove that portable files alone create a reliable organizational memory.

Next test. For one actual team error, record warning disposition, old/new skill diff, name-and-description discovery event, full text visibility, required control execution, human decision and later incident/reporter resolution. A saved review lesson fails if the new plan never loads it, and a loaded lesson fails if no one checks its conclusion.