Guide · data quality
How to verify scraped data is actually correct
The worst scraper failures aren’t crashes. A selector wired to the wrong field returns plausible, wrong data indefinitely — and passes every glance. Verification has to be designed to catch that, and it can’t be done by whoever (or whatever) wrote the scraper.
Why “looks right” fails
Spot checks sample the rows you happened to look at, on the day you looked. They miss the wrong-column wiring, the pagination that silently stopped at page 3, the locale that turned 1.234 into 1,234, and the redesign next month. And self-review compounds it: a person — or a model — reviewing their own extraction approves it. Real validation needs independent checks against the live source, not against the output’s own consistency.
The four checks
- Clean re-run. Execute the whole job again, end to end, in a fresh environment. It must finish without errors — no cached state, no manual nudges. A workflow that only worked once didn’t work.
- Reproducibility. Two runs close in time should produce consistent output. Diffs beyond expected volatility mean the extraction depends on something unstable — ordering, timing, session state — and will drift.
- Resample against the live source. Take a random sample of extracted rows, re-fetch each from the live site or API, and compare values. This catches stale caches, truncated pagination, and off-by-one row alignment that internal checks can’t see.
- Re-prove field semantics. For each column, go back to the live page and confirm the value actually means what the column name claims — that “price” isn’t the crossed-out original, that “rating” isn’t the review count. This is the check that catches the plausible-but-wrong-field failure, and it is the one most pipelines skip.
The independence rule
Whoever grades the scraper must not be able to edit it. The moment the author also judges, the incentives collapse — subtle in humans, structural in AI agents: a model reviewing its own scraper approves it. Separate the roles: the reviewer re-runs the artifact as a black box and scores what comes back. The verdict should be computed from the checks, not argued from an impression.
Common anti-patterns
- Row-count validation. “We got 10,000 rows” says nothing about which 10,000, or whether the columns mean what you think.
- Schema-only validation. Types and nullability pass while every value is from the wrong field.
- Validating once, at build time. Sources rot; checks 1–3 belong in the re-run schedule, not just the launch checklist.
- Letting the builder certify the build. See the independence rule — this is the one that fails silently at scale.
How Zillusion implements this
In Zillusion this method is the architecture, not a checklist: every workflow is re-run by a separate, read-only validator agent in a fresh, container-isolated session — it physically cannot edit what it grades — and a gate computes PASS or FAIL from exactly these four checks. Only a gate-computed PASS promotes the workflow into the robot library, where scheduled re-runs keep re-checking it and diff every new version. The full design is on the verification page.
Frequently asked questions
How do you validate scraped data?
Run four checks: the job re-runs cleanly end to end in a fresh environment; two runs produce consistent output; a random resample of rows re-fetched from the live source matches the extracted values; and each field's meaning is re-proven against the live page. Have the checks graded by a reviewer that didn't write the scraper, and compute the verdict from the checks rather than judgment.
What is the most common silent error in web scraping?
A selector wired to the wrong field: the scraper returns plausible values that mean something else — the discounted price instead of the price, the review count instead of the rating. It passes crashes-only monitoring and casual review, which is why field-semantics re-proving against the live page is a mandatory check.
Why shouldn't the same agent build and approve a scraper?
Because self-review structurally approves its own blind spots — a model reviewing its own scraper says it looks good. Independent verification means the reviewer can run the workflow but cannot modify it, so the verdict reflects what the scraper actually does.
How often should scraped data be re-verified?
On every scheduled re-run, not just at build time. Sources rot: redesigns break selectors, APIs change shape, pagination silently truncates. Re-running the clean-run, reproducibility, and resample checks on schedule — and diffing each version against the last — turns rot into an alert instead of a stale dashboard.