How to turn any website into a dataset

1. Define the record before touching the site

Write down the unit of data you want — one row is a what? (A product, a listing, a country-year, a filing.) List the fields and the type of each. Every later decision — which source, which access path, how to verify — falls out of this definition, and skipping it is how projects end up with data that looks right and answers the wrong question.

2. Find the best access path — scraping the page is the last resort

Data usually has several public paths, and they are not equally good. Check in this order:

1. An official API or bulk download. Stable contract, documented fields, no rendering. Government portals, statistics offices, and many commercial sites have one — often poorly advertised.
2. An undocumented JSON API behind the page. Open browser dev tools → Network tab → filter XHR/fetch while the page loads. Many “scraping” jobs are actually one clean JSON request the page itself makes.
3. Data embedded in the page. JSON-LD blocks, or framework state objects serialized into the HTML — structured data without fighting the layout.
4. The rendered HTML itself. Selectors on tables and lists. Most fragile: a redesign silently breaks them, so everything in steps 4–6 matters more here.

The expensive part of this step is breadth: the right API is rarely the first search result. This is the stage Zillusion industrializes — discovery fans out across 4 search engines, vertical registries, and a 23,000-row API catalog, and scores every candidate on 5 dimensions before anything gets built.

3. Extract with the simplest tool the path allows

An API path needs an HTTP client and pagination handling. A file path needs a download and a parser. Embedded data needs one fetch and a JSON walk. Only the rendered-HTML path needs real scraping machinery — browser automation, selectors, waits. Match the machinery to the path; don’t bring a headless browser to a CSV download.

4. Verify against the live source — not against your own output

The dangerous failure isn’t a crash; it’s a selector wired to the wrong field returning plausible, wrong values forever. Before trusting the data, run four checks: the job re-runs clean; the output is reproducible; a random sample of rows, re-fetched from the live source, matches what you extracted; and each field’s meaning is re-proven against the live page. We wrote this up in detail as a standalone verification guide — it’s the step most pipelines skip.

5. Version the dataset, don’t overwrite it

Store each crawl as a version with a manifest: when it ran, from which source, how many records. Diff each new version against the previous one. Versioning is what turns “the numbers changed” from an argument into a lookup.

6. Monitor for drift — sources rot silently

Schedule re-runs. Alert on failures, but also on drift: record counts jumping, columns going empty, values changing shape. A dataset nobody re-checks is a dashboard quietly going stale.

Doing this with Zillusion

Zillusion runs this exact lifecycle as one conversation: it clarifies the record definition, discovers and ranks the access paths, builds the extraction (preferring APIs over selectors), verifies it in an isolated 4-check gate, then versions the dataset and arms a monitor with webhook or email alerts. In the recorded demo run on our homepage, “Global CO₂ emissions by country” went from sentence to verified, versioned, monitored dataset — 13 sources found, 63 records, gate-computed PASS — in a single session.