In-depth practical guides

Query a public API and retain a verifiable extract

Define the request, check pagination, distinguish errors from absence and preserve the raw response before transformation.

Updated :

A fiber-optic patch panel photographed by Patrick Finnegan in 2002.

A fiber-optic patch panel photographed by Patrick Finnegan in 2002. Documentary context photograph; it does not depict an Alexis Roux ZG assignment or equipment. © Patrick Finnegan · Source · CC BY 2.0. Resized and converted to WebP; cropped display.

How can an extract be reproduced without claiming false completeness?

An API returns a response to a request using a schema at a particular time. This does not mean the whole database was requested or that data stayed unchanged between calls. Retain selection, technical states and received data separately from analysis.

  1. Choose access

    Read official documentation for version, scope, terms, authentication and volume limits. For an entire database, look for an official bulk export; search APIs can cap coverage.

  2. Start with a small request

    Record endpoint, method, parameters, filters and sorting. Prefer stable identifiers to ambiguous names. Never publish keys or tokens in request logs.

  3. Inspect the response

    Record HTTP status, reception time, format and errors. Check fields and units. HTTP 200 may still contain empty data, business errors or a changed schema.

  4. Follow pagination

    Check announced totals and page or cursor mechanics. Count rows and unique identifiers, record duplicates and the stopping condition. Changing data can affect successive pages.

  5. Separate preservation and processing

    Keep raw responses and their digests. Document cleaning, filtering and conversion; link outputs to the original extract. Do not publish personal data or secrets in public logs.

  6. Handle change

    Respect retry guidance for HTTP 429 or unavailable services instead of unlimited retries. A fresh extract is a new observation: compare schema, parameters and scope too.

Minimal request examples

These addresses illustrate syntax; check availability, authentication and terms in the documentation.

Situations and decisions

Typical situations for preparing a check. They do not describe completed assignments or actual observations.

Only one page downloaded

A populated file may cover only part of the selection. Inspect totals, page size and next cursor before publishing a count.

Missing value changed to zero

Keep null, missing fields, empty arrays and zero distinct. Conversion can change averages or rankings.

Same query, new values

Eurostat documents a database holding the latest dataset versions. Retain dated extracts to reproduce earlier calculations.

Record to retain

  • Service, API version, endpoint and secret-free parameters.
  • Timestamp, HTTP state, format, pages and stopping rule.
  • Raw responses, unique identifiers and file digests.
  • Transformations, units, exclusions and completeness limits.

Official references

Related source profiles

Related protocols

Checks before reaching a conclusion

  • Can selection be reconstructed without a secret?
  • Do published counts account for pages and duplicates?
  • Are raw responses separate from cleaned data?

Frequently asked questions

Does HTTP 200 prove completeness?

No. Inspect body, filters, totals and pagination.

Must the whole database be collected?

No. Choose the useful scope and document its limits.

Does a digest guarantee the figures?

It compares preserved file bytes. Methods and data quality require their own checks.