Anthropic webapp-testing: one local checkout trial

Tested and written by Fred at SkillsTested · Published 2026-10-12 · Updated 2026-10-12

Can this Skill help an agent run a local web app and capture a browser failure? In one isolated Codex run, it loaded Anthropic's pinned webapp-testing instructions, started our synthetic checkout, and produced a replayable Playwright check. The check caught a deliberately seeded discount bug. We supplied the expected result in the prompt, so this is a workflow demonstration, not a blind bug-finding benchmark.

The task and source check

We used a small local checkout with an asynchronous plan list: Starter costs $12 and Team costs $24. Its interface says promo code SAVE10 gives 10% off. Selecting Team should show $24.00; applying the code should show $21.60. The seeded defect displays “10% discount applied” but leaves the total at $24.00. The prompt named the expected $21.60 and allowed the agent to use a relevant installed project Skill without naming webapp-testing.

Before installation, we pinned the official repository commit and reviewed the Skill, its Apache 2.0 license, three examples, and server helper. The helper runs a caller-supplied server command via a shell; use only a command you trust. This trial used a local Python fixture, no credentials, and blocked non-local browser requests. The reviewed helper did not require an external API for this job.

What the agent did

1. Installed webapp-testing in an isolated project's .agents/skills/. On a fresh natural-language request, Codex selected and read that Skill. This is one observed activation, not a reliability rate.
2. Read the bundled server helper's help, started the local fixture, and wrote a Python Playwright script using system Chrome. The first attempt timed out on a status locator; it changed to the page's #message locator and reran.
3. Selected Team and observed $24.00 as expected. After entering SAVE10, it saw the success message but still observed $24.00, rather than $21.60. It saved JSON findings and a screenshot.
4. We independently reran the generated script with the bundled server helper. It reproduced the same two observations and exited 1 to mark the checkout check as failed. Fixture file hashes matched their pre-run values.

Use it for, and where this evidence stops

This workflow is useful when a local app needs a reproducible browser check, screenshot, and explicit expected-versus-observed result. In this trial, the agent generated the check and corrected one selector problem before recording the fixture's failure. The replay bundle contains the synthetic app, script, findings, and setup instructions. It reproduces the scripted browser finding, not Codex's Skill selection; install Playwright 1.63.0 and use system Chrome on macOS (or adapt the executable path on another system).

We did not run a no-skill comparison, test Claude Code activation, assess complex apps, or measure repeated trigger reliability. The test was guided toward a known defect. A local sandbox initially denied the listener, so the server/browser step needed a scoped retry. No external browser requests or account access were observed. For activation context, see how installed skills are selected; inspect the pinned upstream folder before using a later revision.