Can Coding Agents Repair the Same Broken Web Project?

Result

In this small repair task, both returned projects passed all 10 automated validator checks. The frozen starting project passed only 2/10. In both final archives, the benchmark validator and package.json were unchanged, while the actual repair was confined to index.html, css/site.css and js/app.js.

This result does not establish that one coding agent is generally better than another. It establishes something narrower and reproducible: for this fixture and acceptance test, both submitted artifacts satisfied every automated check without modifying the test harness.

Frozen fixture

The benchmark is intentionally small: one static HTML page, one stylesheet, one JavaScript file and a Node-based validator. The page contains defects across responsive behavior, accessibility state and a simple JavaScript interaction.

The fixed task required the agent to:

1. hide and toggle the mobile navigation at 700px or narrower;
2. expose aria-controls="mainNav" and synchronized aria-expanded state;
3. collapse three feature cards to one column without fixed/minimum-width overflow;
4. make the existing demo button update #demoStatus to Demo completed.;
5. preserve desktop navigation and the three-column desktop grid;
6. add no new runtime dependency;
7. make the smallest reasonable changes and leave the validator untouched; and
8. run npm test until all checks passed.

Before an agent repair, the frozen fixture produced:

PASS | HTML has a single H1
FAIL | Menu button exposes expanded state
FAIL | Menu button controls navigation
FAIL | Navigation has hidden mobile default
FAIL | Open mobile navigation is displayed
FAIL | Responsive feature grid becomes one column
FAIL | Feature cards do not force 420px minimum
FAIL | Demo JS targets existing status element
FAIL | Menu JS updates aria-expanded
PASS | Menu toggle uses is-open
RESULT: 2/10 checks passed

Method

We created one frozen benchmark package and kept the task text and validator fixed. Each clean run was intended to start from that same broken fixture. The final returned project was then checked with npm test and compared against the frozen source at file level.

The validator checks observable source conditions tied to the acceptance criteria. It is not a browser automation suite and it does not measure visual quality, semantic code quality, security or performance.

There was also an excluded Claude Code attempt: Claude was accidentally pointed at the already repaired Codex copy. It correctly reported that all 10 checks already passed and made no changes. Because that input was not the frozen 2/10 fixture, that attempt is not counted as a benchmark run. The Claude result reported below is the separate result archive produced from the clean benchmark workflow.

Codex result

The Codex result passed 10/10 checks. File comparison shows changes to exactly three product files: index.html, css/site.css and js/app.js. tests/validate.mjs and package.json match the frozen fixture.

The patch added the two ARIA attributes to the menu button, corrected the mobile navigation's default/open display states, changed the mobile feature grid to one column, removed the fixed 420px card minimum by replacing it with min-width: 0, synchronized aria-expanded in JavaScript, and corrected the demo status selector from #demo-state to #demoStatus.

RESULT: 10/10 checks passed

Claude Code result

The Claude Code result also passed 10/10 checks and changed the same three product files while leaving tests/validate.mjs and package.json unchanged.

Its HTML and JavaScript repairs are functionally the same as the Codex patch: the menu gains aria-controls and initial aria-expanded, the click handler synchronizes expanded state, and the demo handler targets #demoStatus. In CSS, it corrected the mobile navigation and one-column grid but removed the broken min-width: 420px declaration instead of replacing it with min-width: 0.

RESULT: 10/10 checks passed

What differed between the patches?

The final implementations are nearly identical. The only substantive source difference between the two submitted projects is the feature-card width rule:

/* Codex */
.feature-card { ... min-width: 0; }
/* Claude Code */
.feature-card { ... }

Both forms satisfy this fixture's validator and the stated requirement not to force the cards to a 420px minimum width. The benchmark therefore gives us no evidence for choosing one implementation as superior in this project.

That is useful in its own right: the same acceptance criteria can lead independent agents to almost the same patch while still leaving small implementation choices open.

Reproduce or inspect the test

The exact artifacts used for this record are published with the page:

• Frozen benchmark package — broken fixture, fixed task and validator.
• Codex result archive — submitted repaired project.
• Claude Code result archive — submitted repaired project.

After extracting the frozen package, run the baseline from its project directory:

npm test

A faithful repeat should preserve the frozen task and validator. Keep each agent run in a separate copy so that one repaired project cannot become another agent's starting state.

Limitations

This is one deliberately small fixture, not a broad coding benchmark. Ten source-oriented checks are enough to verify the defined task, but they do not substitute for browser automation, manual accessibility testing, cross-browser testing or review of a production application.

The experiment did not retain a complete comparable execution trace for both clean runs, so this page does not compare command count, elapsed time, token use, cost or number of reasoning steps. Exact underlying model versions were also not recorded in the benchmark artifacts, so the results are attributed only to the user-reported Codex and Claude Code tool runs.

The first Claude attempt was invalid because its starting project had already been repaired; it is explicitly excluded rather than silently treated as a successful run. Future repetitions should record tool/model version, start timestamp, end timestamp and full agent transcript before any files are modified.