In-depth practical guides

Query a public API and retain a verifiable extract

Define the request, check pagination, distinguish errors from absence and preserve the raw response before transformation.

Updated :

Laptop and tablet showing a schematic API response and a row to check before import. Illustrative scene.

AI-generated illustration

How can an extract be reproduced without claiming false completeness?

An API returns a response to a request using a schema at a particular time. This does not mean the whole database was requested or that data stayed unchanged between calls. Retain selection, technical states and received data separately from analysis.

  1. Choose access

    Read official documentation for version, scope, terms, authentication and volume limits. For an entire database, look for an official bulk export; search APIs can cap coverage.

  2. Start with a small request

    Record endpoint, method, parameters, filters and sorting. Prefer stable identifiers to ambiguous names. Never publish keys or tokens in request logs.

  3. Inspect the response

    Record HTTP status, reception time, format and errors. Check fields and units. HTTP 200 may still contain empty data, business errors or a changed schema.

  4. Follow pagination

    Check announced totals and page or cursor mechanics. Count rows and unique identifiers, record duplicates and the stopping condition. Changing data can affect successive pages.

  5. Separate preservation and processing

    Keep raw responses and their digests. Document cleaning, filtering and conversion; link outputs to the original extract. Do not publish personal data or secrets in public logs.

  6. Handle change

    Respect retry guidance for HTTP 429 or unavailable services instead of unlimited retries. A fresh extract is a new observation: compare schema, parameters and scope too.

Minimal request examples

These addresses illustrate syntax; check availability, authentication and terms in the documentation.

Reading workshop

A revalidation log, not a new empty extraction

The DEMO HTTP exchanges are fictional. This section contacts no server and performs no retries.

Fictional data and records for learning: none of these results comes from a live service query.

Separate the copy, validation and a request limit

The scenario client retains a representation for area=A with its ETag. The fictional retry policy caps attempts at two. It guarantees no service availability.

  1. Keep the initial copy

    Fictional exchange
    GET area=A → 200, body and ETag "demo-v1"
    Retained state
    Body linked to the exact request
    Scenario decision
    Keep filters, version and representation together.
  2. Revalidate the same selection

    Fictional exchange
    Same request with If-None-Match → 304 without a body
    Retained state
    A matching copy is available
    Scenario decision
    Reuse that copy; do not record a zero-row extraction.
  3. Change the selection

    Fictional exchange
    New request area=B
    Retained state
    The A copy does not document B
    Scenario decision
    Handle separately even if ETag text looks similar.
  4. Reach a limit

    Fictional exchange
    429; Retry-After: 15
    Retained state
    This attempt failed; not a zero result
    Scenario decision
    Wait as indicated; stop and log when the two-attempt cap is reached.

Which log correctly separates collection states?

Open each proposition to compare its diagnosis. More checks may still be needed.

304: a new empty selection

No. This scenario still uses the copy of the same selection. An absent body does not become an empty table.

Copy A data into the B extraction

No. That silently changes observation scope. Parameters belong in the record.

Retain the A-copy reference and separately record the limit-related stop

Yes. The log must identify data used and attempts that supplied no new extraction.

Fictional client record

Selection: GET /demo?area=A
Copy: body retained; ETag="demo-v1"
Condition: If-None-Match="demo-v1"
Response: 304; reference to A copy
Other attempt: 429; Retry-After=15 seconds
DEMO policy: two attempts maximum, then stop
Republication rights: check separately

The two-attempt cap belongs to this example, not the RFCs. Without a usable copy, do not invent content; plan another retrieval under the service policy.

Documentation of the rules, separate from fictional examples

Compare other situations and exercises

Only one page downloaded

A populated file may cover only part of the selection. Inspect totals, page size and next cursor before publishing a count.

Missing value changed to zero

Keep null, missing fields, empty arrays and zero distinct. Conversion can change averages or rankings.

Same query, new values

Eurostat documents a database holding the latest dataset versions. Retain dated extracts to reproduce earlier calculations.

Worked exercise

Status 200 with an application error

Fictional example: the values and documents teach the method and do not describe a real organisation.

This fictional API’s contract requires success=true before processing. Check the HTTP response, application state and presence of results separately.

If the table exceeds the screen, scroll horizontally; arrow keys work when the region is focused.

Starting information
ItemObserved information
HTTP status200
Application indicatorsuccess=false
Message receivedMessage: quota reached
Results arrayNo array supplied

Your checks

  1. Check HTTP status and application indicator separately.
  2. Verify the expected schema is present.
  3. Retain the error without publishing a false count.
Read the answer

Answer and decision

Transport responded, but collection failed under this contract. No zero count is established. Mark extraction failed and follow the documented retry rule; do not manufacture an empty table to continue calculation.

If the case changes

If a successful response truly has an empty array, zero can be established for the stated query and scope. Replay with success=true but a JSON body truncated before completion: status and flag still do not suffice. Reject the incomplete batch, retain the recovery point and verify that no partial value replaced the last validated extract.

Record to retain

Keep a secret-redacted query, status, error, contract and retry state.

Documentation reference : IETF — RFC 9110, 200 OK · Consulted on

DOCUMENTED REFERENCE

Wikidata: truthy selects a rank, not certified truth

Truthy is a technical projection. For one entity-property pair, preferred statements exclude normal ones from that projection; otherwise normal statements are selected. This is neither automatic source voting nor latest-date selection.

Wikidata: truthy selects a rank, not certified truth
ReferenceValue or formWhat it identifies
PreferredRankPreferredAll preferred statements are selected when present for that entity and property. Preferred does not ensure uniqueness.
NormalRankNormalNormal statements are selected only if that pair has no preferred statements. Several normal values may be legitimate.
DeprecatedRankDeprecatedDeprecated statements are excluded from truthy, but can remain in full statements to document errors or rejected values.
wdt:Best non-deprecated rankThe direct path uses best non-deprecated ranks without qualifiers. Extracted values are not automatically current, complete or supported.
wikibase:BestRankFull statement selectionThis type selects best-rank full statement nodes, retaining access to qualifiers and references.
ScopeEntity + propertyThe rule is per entity and property. Preferred population does not select a foundation date, name or another property.

Implication for the record

If all statements are deprecated, do not invent a fallback to them. For disagreement or history read full statements and retain ranks. Missing direct-query values may be filtered by rank rather than absent from the database.

Is a wdt: value necessarily the latest?

No. Rank and temporal qualifiers are separate mechanisms. Check point in time and rank practice before calling it latest; multiple preferred statements may coexist.

Primary documentation

Documentation consulted on .

DOCUMENTED REFERENCE

World Bank API: distinguish three dates and recover the unit

A JSON response contains more than a value. The real excerpt below concerns Finnish density in 2023. Its header reports a WDI update on 8 October 2026; this snapshot was retrieved on 23 October. Those three dates answer different questions.

date2023lastupdated2026-10-08extracted2026-10-23
World Bank API: distinguish three dates and recover the unit
ReferenceValue or formWhat it identifies
date2023Observation year. It does not become 2026 because the request ran in 2026.
lastupdated2026-10-08Update date reported in the source header. It does not by itself certify revision of this particular value.
extracted2026-10-23Date this local snapshot was assembled; it replaces neither the observation period nor the source date.
unit""The record has an empty unit field. The EN.POP.DNST definition still specifies people per km² of land: empty does not mean dimensionless.
decimal / value0 / 18.3705454665088The decimal field is zero while value has fractional digits. Preserve the supplied value; do not interpret this field as an error margin.
pages / total1 / 4The four observations fit on one response page. Another request may be paginated: check pages and total before treating collection as complete.

Exact request · density

https://api.worldbank.org/v2/country/FRA;DEU;ESP;FIN/indicator/EN.POP.DNST?date=2023&format=json&per_page=100

Preserved header and FIN observation

[
    {
        "page": 1,
        "pages": 1,
        "per_page": 100,
        "total": 4,
        "sourceid": "2",
        "lastupdated": "2026-10-08"
    },
    [
        {
            "indicator": {
                "id": "EN.POP.DNST",
                "value": "Population density (people per sq. km of land area)"
            },
            "country": {
                "id": "FI",
                "value": "Finland"
            },
            "countryiso3code": "FIN",
            "date": "2023",
            "value": 18.3705454665088,
            "unit": "",
            "obs_status": "",
            "decimal": 0
        }
    ]
]

Implication for the record

This is a dated API snapshot, without automatic refresh on page load. A shared country code does not prove identical territorial coverage across series. To reproduce the analysis, retain the full request, definitions and missing values; do not turn null into zero.

Primary documentation

Documentation consulted on .

DOCUMENTED REFERENCE

Form parameters: preserve spaces, plus signs and duplicate names

A parameter list is an ordered sequence of name-value pairs. The same name may occur twice. This tool applies application/x-www-form-urlencoded through URLSearchParams to two single-line values, then parses the result; it submits no form.

tag → C++ & IAtag → ététag=C%2B%2B+%26+IA&tag=%C3%A9t%C3%A9getAll("tag") → ["C++ & IA", "été"]
Form parameters: preserve spaces, plus signs and duplicate names
ReferenceValue or formWhat it identifies
U+0020+A U+0020 space is serialized as + in this format. This is not a universal rule for every URL component.
+%2BA plus sign belonging to the value becomes %2B. Pasting C++ directly into an already encoded string would make its two plus signs parse as spaces.
&%26An ampersand within a value becomes %26, so it does not introduce an extra parameter.
=%3DAn equals sign within a value becomes %3D; the structural name-value separator remains =.
été%C3%A9t%C3%A9Non-ASCII characters are encoded from UTF-8. Percent escapes represent bytes, not a count of displayed letters.
append / get / getAll / setordered pairsappend retains a new occurrence. get returns the first value; getAll returns all values in order. set replaces the first and removes the other occurrences of that name.

Explore two parameters with the same name

Two demonstration values calculated on this page.

Two pairs preserved in order.

Serialization

tag=C%2B%2B+%26+IA&tag=%C3%A9t%C3%A9

Values parsed with getAll

["C++ & IA","été"]

First value with get

"C++ & IA"

Implication for the record

Encoding is neither encryption nor HTML sanitization. This demonstration does not simulate form-control selection, multipart files or form line-break normalization. The server must explicitly define duplicate-parameter handling; converting the list to an object too early can lose values.

Primary documentation

Documentation consulted on .

Record to retain

  • Service, API version, endpoint and secret-free parameters.
  • Timestamp, HTTP state, format, pages and stopping rule.
  • Raw responses, unique identifiers and file digests.
  • Transformations, units, exclusions and completeness limits.

Official references

Related source profiles

Related protocols

Checks before reaching a conclusion

  • Can selection be reconstructed without a secret?
  • Do published counts account for pages and duplicates?
  • Are raw responses separate from cleaned data?

Frequently asked questions

Does HTTP 200 prove completeness?

No. Inspect body, filters, totals and pagination.

Must the whole database be collected?

No. Choose the useful scope and document its limits.

Does a digest guarantee the figures?

It compares preserved file bytes. Methods and data quality require their own checks.

How to recover an interrupted collection?

Keep the last step confirmed, parameters and continuation references. The recovery should check the duplications and possible changes of the data rather than suppose a simple addition of answers.

DOCUMENTED REFERENCE

HTTP writes: check the version before replacing a document

Two editors may read one version and submit different corrections. A precondition lets the server reject a write based on an older representation. These examples are fictional: v1 and v2 are teaching tags, not observations from a service.

GET → "v1"ETag: "v2"PUT → 412If-Match: "v1" ≠ ETag: "v2"

Read → concurrent edit → rejected write: fictional example.

HTTP writes: check the version before replacing a document
ReferenceValue or formWhat it identifies
PUT · If-Match: "v1"ETag: "v2" → 412The document changed after it was read. Strong comparison fails for this PUT: do not replace the current version. Read it again and review the differences.
PUT · If-Match: "v2"ETag: "v2" → ✓The strong tags match, so this precondition permits normal processing to continue. It guarantees neither status 200 nor success of other checks.
PUT · If-Match: W/"v2"ETag: "v2" → 412The opaque text matches but one tag is weak. If-Match requires strong comparison and fails. Removing W/ on the client does not create a strong validator.
PUT · If-None-Match: *Present → 412 / Absent → ✓For a write, the asterisk requires that no current representation exist. An existing representation makes the condition fail; absence permits processing to continue.
GET · If-None-Match: W/"v2"ETag: "v2" → 304For GET or HEAD, If-None-Match uses weak comparison. A match leads to 304: reuse the retained copy without a new response body.
428Precondition RequiredThe server requires a conditional request. A missing precondition differs from 412, where a supplied condition was false. A 428 response must not be cached.

Try a precondition

Change the method, state and condition. Enter a quoted tag, a comma-separated tag list or * alone. Evaluation stays in your browser. With * or an absent representation, the current ETag is not used.

412: precondition false. Do not perform the requested operation.

Local teaching simulation: no API request is sent and no data is saved.

Implication for the record

This exercise evaluates one precondition for one resource before acting. It assumes an otherwise acceptable request and excludes the special case of an operation already completed successfully. Authentication, authorization, document validation and business rules still apply. An API must check and apply the change atomically to prevent lost updates.

Primary documentation

Documentation consulted on .

DEEPER READING / PRACTICE

Preserve a reproducible query

A API provides useful data, but the result depends on settings and pageing. The collection must keep the complete request and the conditions necessary to understand the answer.

Updated on .

Priority checks

Release URL, parameters, status, format and page. Control the filters, volume limits and returned errors without exposing secrets in the journal.

Interpretation pitfalls

A successful answer may contain only one page of results. Repeating the request later may provide a different state if the data has changed.

Working record to retain

The collection journal combines request, date, preserved answer, coverage and limit. It specifies how to recover or check the extraction.