In-depth practical guides

Keep a reproducible version of public data

Connect source file, dates, schema and transformations: retain data that can be located, compared and reused without confusing an update, correction or change of scope.

Updated :

File and table checks at an organised workstation.

AI-generated illustration

Can another reader reconstruct the result?

A link to a changing database does not identify the data actually used. Corrections may revise old series, exports may lose leading zeros and new columns may change the meaning of a total. A useful version explains the starting material and processing path. The aim is reproducibility of the described work, not a universal guarantee of source quality.

  1. Define scope

    Name producer, dataset, territory, population, period and filters. Separate observation, publication and download dates. An export received today may describe an old period. State the question and exclusions important to interpretation.

  2. Retain the actual input

    Record URL or public parameters, retrieval date and received file when permitted. Assign a version identifier and compute a file digest. A digest helps distinguish files; it does not establish value accuracy. Keep license information with the version.

  3. Describe the schema

    Document fields, types, units, keys and missing values. For CSV, inspect delimiter, encoding, quotes and line breaks in the actual export. Preserve identifiers requiring leading zeros as text. Distinguish zero, missing and not-applicable values.

  4. Trace transformations

    Record filtering, normalization, joins, conversions and deduplication rules in order. Separate raw input from prepared output. Retain join keys and unresolved cases. Similar names alone do not justify merging entities. Document choices instead of hiding them in the final file.

  5. Compare versions

    Measure added, removed and changed rows using a stable key where available. Also inspect structure, units and definitions. Row-count changes do not establish changes in the underlying phenomenon: filters, coverage or producer revisions may explain them.

  6. Prepare reuse

    Associate the result with its source version, steps and limits. Check links leading back to the producer and context. Plan a new collection with comparison criteria. Retain an older version separately when it supports a published conclusion, respecting conditions and the data it contains.

Reading workshop

A revised value does not rewrite what was known

The DEMO dataset is fictional. Its versions separate historical analysis from analysis using corrected data.

Fictional data and records for learning: none of these results comes from a live service query.

80 in April, 82 after correction in June

Both files contain the same key and period. The fictional producer publishes a correction note. A later catalogue-record edit leaves the CSV unchanged.

If the table exceeds the screen, scroll horizontally; arrow keys work when the region is focused.

80 in April, 82 after correction in June
Fictional stateAvailable recordWhat changesUse to disclose
v1 available in AprilValue 80 for the selected keyPublished state before correctionTo describe what was accessible in April, cite v1.
v2 published in JuneCorrected value 82 and revision notePast observation corrected, not a new eventFor revised analysis, cite v2 and disclose the correction.
Record edited after v2Description clarified, CSV unchanged in this caseMetadata onlyDo not infer a new value from the record date.
Both analyses retainedHistorical result with v1; revised result with v2Time-related question and source versionDo not silently mix versions.

Should the April result be silently replaced?

Open each proposition to compare its diagnosis. More checks may still be needed.

Yes: only the newest file deserves a record

No. The old version may be needed to understand the conclusion published then.

No: state question, version and correction explicitly

Yes. Separate what was known in April from June’s revised analysis; link outputs to their input.

An identical hash certifies data quality

No. It concerns bytes compared with a reference, not whether a value is accurate or representative.

Fictional versioning record

Dataset: DEMO; key and period unchanged
v1: available in April; value=80
v2: published in June; corrected value=82
Reason: documented correction by fictional producer
Later record: description only, CSV unchanged
Outputs: historical linked to v1; revised linked to v2

The record keeps both analytical questions. Hashes identify the files used without making a retained version a true version.

Documentation of the rules, separate from fictional examples

Compare other situations and exercises

Identifiers change after opening

Software treats a column as numeric and removes leading zeros. Return to the raw export, require text for the key and check joins. Reformatting an already altered value may not reconstruct the original identifier.

A historical series is revised

The new file also replaces earlier values. Compare by period and key, retain both versions and read producer notes. Separate revised observations from newly added ones before describing a trend.

The file grows without a measured improvement

Coverage now includes more territories or categories. Compare the common scope, then describe the extension separately. Record field changes and missing values rather than assuming every row is comparable.

Separate absence from modification

Fictional example: A changes from 10 to 11, B disappears and C arrives. With id as key, the report counts one added key, one removed key and one modified shared record. A new column is listed separately; no previous value is invented. Reordering alone produces no modification.

Worked exercise

Separate revision from addition in a rise

Fictional example: the values and documents teach the method and do not describe a real organisation.

Keys are unique, values additive and the dictionary unchanged. Nothing is deleted; “absent” means no such key in that version.

If the table exceeds the screen, scroll horizontally; arrow keys work when the region is focused.

Starting information
KeyVersion V1Version V2
A1010
B2025
CAbsent5

Your checks

  1. Compare rows by exact key.
  2. Separate B's revision from C's addition.
  3. Reconcile the difference in totals.
Read the answer

Answer and decision

V1 totals 30 and V2 40: +10. On shared keys, B's revision explains +5; new key C adds +5. A is unchanged. The total's 33.33% rise should not be attributed entirely to growth among existing objects.

If the case changes

If units or definitions change in V2, pause this decomposition until comparability is checked.

Record to retain

Keep exports and hashes, dictionaries, added/changed keys and reconciliation formula.

Record to retain

  • Producer, URL, public parameters, license and separate dates.
  • File version, digest, schema, units and missing-value conventions.
  • Ordered transformations, join keys and unresolved cases.
  • Version comparison, reconstructed result and interpretation limits.

Official references

Related protocols

Checks before reaching a conclusion

  • Does an identifier preserve its exact characters after import and export, including leading zeros?
  • Does the comparison separate changes in values, schema and coverage instead of reducing every difference to a row count?
  • Can another reader reconstruct the output from the retained version and the documented transformation order?

Frequently asked questions

Should every export be retained?

Keep versions needed for the work and published conclusions under a defined retention rule. Avoid purposeless copies and respect storage and reuse conditions.

Does a digest guarantee quality?

No. It identifies file content and helps detect a change; it does not validate provenance, interpretation or values.

Can rows be merged by name?

Check keys, context and namesakes first. A matching rule must handle ambiguous cases and record merges.

How should corrected data be displayed?

Identify the version and relevant correction, separating the revised result from the previously published one. Do not present a revision as a new observation.

Why reject repeated keys?

Two rows with the same identifier make matching ambiguous. Check their grain, period and origin, then define a compound key if needed. Do not remove a row before establishing whether it is a duplicate or a separate observation.

What to keep with a version of data?

The file, its origin, its date, its hash and the useful documentation. Adding the processing rules and the reference of the products results facilitates subsequent reproduction and comparison.

DOCUMENTED REFERENCE

PROV provenance: connect version, process and responsibility

W3C PROV-O describes how documents or data were produced. Arrows follow property direction, not a left-to-right timeline. E0, A and E1 are teaching labels, not real resources.

E0 · EntityA · ActivityE1 · EntityusedwasGeneratedBywasDerivedFrom
PROV provenance: connect version, process and responsibility
ReferenceValue or formWhat it identifies
prov:EntityData/document in a described stateEntity represents data or documents in a described state. Distinguish versions; a changing URL alone is insufficient.
prov:ActivityProcess over timeActivity is a process over time acting on entities, not its output object.
prov:AgentPerson, organization or software agentAgent bears responsibility. wasAttributedTo links entity to agent; wasAssociatedWith links activity to agent, not proof of licensing or exhaustive legal responsibility.
prov:usedActivity → Entityused goes activity to entity: A uses E0.
prov:wasGeneratedByEntity → ActivitywasGeneratedBy goes output entity to activity: E1 was generated by A, not a version-to-version relation.
prov:wasDerivedFromEntity → EntitywasDerivedFrom goes result entity to source entity; it does not provide formula, weights or fidelity proof.
prov:wasRevisionOfEntity → EntityRevision is a specific derivation retaining substantial original content; citing a document is not necessarily revising it.
prov:wasQuotedFromEntity → EntityA passage is repeated from a larger source. Retain its exact wording and location; the relation alone does not locate the words.

Quotation, translation or revision?

Four fictional cases show why a link must describe the actual transformation. The result is the subject of each relation; its source or agent is on the right.

A copied passage

quote → prov:wasQuotedFrom → source-v1

A passage is repeated from a larger source. Retain its exact wording and location; the relation alone does not locate the words.

A translation

translation → prov:wasDerivedFrom → source-v1

The translation is a new derived entity. Retain source and target languages, version and fidelity checks; translated text must remain distinct from the original passage.

A document correction

source-v2 → prov:wasRevisionOf → source-v1

The corrected version retains substantial content from its predecessor. Keep both states and the changes; this relation does not determine which version is accurate.

Attribution of the result

analysis → prov:wasAttributedTo → editor

The entity is attributed to an agent. For responsibility for a process, use Activity → prov:wasAssociatedWith → Agent. Attribution does not establish reuse rights.

Provenance describes an asserted chain. It makes neither a translation independent evidence nor attribution validation of the content.

Implication for the record

Describe known inputs, output, activity and agent without inventing gaps. Provenance records a transformation assertion, not automatic correctness, rights or authenticity. Preserve versions and source references.

Does wasDerivedFrom suffice to reproduce an indicator?

No. Exact versions, filters, formula, parameters and missing-value conventions are also needed; one source can support different methods.

Primary documentation

Documentation consulted on .

DOCUMENTED REFERENCE

Combining fragments: coverage, gaps and representation identity

Adding received lengths can conceal an incomplete transfer: overlap counts the same positions repeatedly. Compute the union of the intervals and compare it with the intended representation. This geometric check remains separate from byte identity and provenance.

Combining fragments: coverage, gaps and representation identity
ReferenceValue or formWhat it identifies
0-399,300-599,700-999Σ = 1000; ∪ = 900For L = 1,000, fragments 0–399, 300–599 and 700–999 total 1,000 declared bytes. Their union covers only 900 positions: 600–699 is missing and 100 byte receipts are redundant.
0-499,500-999∪ = 1000Adjacent intervals merge without a gap: 0–499 and 500–999 cover 1,000 positions. Receiving the same byte three times adds two redundant receipts, rather than two distinct overlapping positions.
Content-Rangefirst ≤ last < LA response Content-Range must have an end at least equal to its start; a known total length must be strictly greater than that end. An unknown length marked * cannot establish full coverage.
ETagstrong validatorFragments must belong to the same representation. RFC 9111 requires a shared strong validator and the HTTP recombination conditions when a cache combines partial responses. A matching URL or length does not replace this check.
If-Rangestrong comparisonIf-Range uses strong ETag comparison. A client must not send a weak ETag in it; a failed condition causes Range to be ignored and requests the full transfer. A date is allowed only under HTTP’s strong-validator conditions.

Check fragment coverage

Original teaching examples; no measurement of a real server.

L: integer from 1 to 1,000,000,000,000. Enter 1–20 inclusive a-b intervals, one per line, within the representation. Blank lines are ignored; spaces and tabs around lines are allowed. At most 2,000 characters; no HTTP header.

Calculated locally in your browser; entered data is neither transmitted nor stored.

Calculation updated.

Sum of declared lengths
1000
Distinct covered positions
900
Redundant byte receipts
100
Missing positions
100

Arithmetic coverage only: representation identity and byte integrity still require verification.

Representation coverageLeft-to-right bar: positions 0 through L − 1. Solid segments are covered; hatched segments are missing. The table gives exact endpoints; very small ranges may be imperceptible at this scale.0999
Left-to-right bar: positions 0 through L − 1. Solid segments are covered; hatched segments are missing. The table gives exact endpoints; very small ranges may be imperceptible at this scale.
Union of fragments
Inclusive intervalLength
0–599600
700–999300
Missing intervals
Inclusive intervalLength
600–699100

Calculation rules

sum = Σ(lastᵢ − firstᵢ + 1)
unique = |∪ intervals|
redundant = sum − unique
missing = L − unique
Eight coverage cases
Eight coverage cases
LDeclared fragments (a-b, one per line)Sum of declared lengthsDistinct covered positionsRedundant byte receiptsMissing positionsMissing intervals
10000–399; 300–599; 700–9991000900100100600–699
10000–499; 500–9991000100000None
10000–999; 0–9992000100010000None
100–0; 0–0; 0–031291–9
109–9; 0–022081–8
102–7; 3–486240–1; 8–9
10–01100None
10000000000000–999999999998; 999999999999–9999999999991000000000000100000000000000None

Implication for the record

Coverage assumes declared positions in one representation. This tool reads no files and checks no ETag, Content-Encoding, cache or digest. It does not establish equal bytes within an overlap. Complete coverage is not an integrity certificate.

Does a matching ETag authenticate a document?

No. An ETag is an HTTP validator, not necessarily a cryptographic digest or an author’s signature. Also check the variant, lengths and actual received bytes; record provenance and integrity checks separately.

Primary documentation

Documentation consulted on .

DOCUMENTED REFERENCE

JSON Patch: follow operations on the current document

A JSON Patch list is evaluated in order. Each path addresses the document produced by previous operations, not a permanent snapshot of the initial document. These nine original walkthroughs show calculation states and stopping conditions.

JSON Patch: follow operations on the current document
ReferenceValue or formWhat it identifies
addinsert / replaceadd inserts at an array index and shifts later elements; a final hyphen appends after the last element. In an object, add also replaces an existing member. The parent container must exist.
remove / replaceexisting targetremove and replace require an existing target. Removing an array element shifts higher indexes left, so the same path can subsequently select a different element.
moveremove → addmove first removes the from value, then adds it at path in the modified document. Moving to a later array position requires using the resulting indexes. Moving into a proper descendant is forbidden.
copy / testvalue / equalitycopy adds the referenced value without linking both locations. test compares a value using JSON rules, writes nothing and stops evaluation on failure. Object member order is not significant for equality.
HTTP PATCHatomicAn error stops the sequence. In HTTP PATCH, all changes must be applied atomically: a failed patch applies no changes. The reader’s intermediate states explain the calculation; they are not committed writes.
Content-Typepatch formatContent-Type distinguishes application/json-patch+json from application/merge-patch+json. PATCH support alone does not guarantee both formats. An HTTP precondition such as If-Match can prevent an update against a stale version; test does not automatically replace it.

Read a JSON Patch sequence

Original teaching examples; no real document is modified.

Choose a walkthrough and step. Each operation addresses the preceding result. On failure, the reader shows the initial target again to illustrate HTTP atomicity; later operations are not executed.

Predefined walkthrough reader; no arbitrary patch is executed here.

Initial state · 0 / 2

Initial target

{
    "items": [
        "A",
        "B",
        "C"
    ]
}

Operation

No operation at this step.

Calculation state

{
    "items": [
        "A",
        "B",
        "C"
    ]
}
Nine walkthroughs with all their states
Removal and shifting indexes
  1. Initial state

    No operation at this step.

    Calculation state

    {
        "items": [
            "A",
            "B",
            "C"
        ]
    }
  2. Intermediate state, not committed

    Operation

    {
        "op": "remove",
        "path": "/items/0"
    }

    Calculation state

    {
        "items": [
            "B",
            "C"
        ]
    }
  3. Sequence succeeded

    Operation

    {
        "op": "replace",
        "path": "/items/1",
        "value": "X"
    }

    Calculation state

    {
        "items": [
            "B",
            "X"
        ]
    }
Insertion and append
  1. Initial state

    No operation at this step.

    Calculation state

    [
        "A",
        "C"
    ]
  2. Intermediate state, not committed

    Operation

    {
        "op": "add",
        "path": "/1",
        "value": "B"
    }

    Calculation state

    [
        "A",
        "B",
        "C"
    ]
  3. Sequence succeeded

    Operation

    {
        "op": "add",
        "path": "/-",
        "value": "D"
    }

    Calculation state

    [
        "A",
        "B",
        "C",
        "D"
    ]
Null value then deletion
  1. Initial state

    No operation at this step.

    Calculation state

    {
        "note": "old",
        "keep": true
    }
  2. Intermediate state, not committed

    Operation

    {
        "op": "replace",
        "path": "/note",
        "value": null
    }

    Calculation state

    {
        "note": null,
        "keep": true
    }
  3. Sequence succeeded

    Operation

    {
        "op": "remove",
        "path": "/note"
    }

    Calculation state

    {
        "keep": true
    }
Copy without a link
  1. Initial state

    No operation at this step.

    Calculation state

    {
        "source": {
            "n": 1
        }
    }
  2. Intermediate state, not committed

    Operation

    {
        "op": "copy",
        "from": "/source",
        "path": "/backup"
    }

    Calculation state

    {
        "source": {
            "n": 1
        },
        "backup": {
            "n": 1
        }
    }
  3. Sequence succeeded

    Operation

    {
        "op": "replace",
        "path": "/source/n",
        "value": 2
    }

    Calculation state

    {
        "source": {
            "n": 2
        },
        "backup": {
            "n": 1
        }
    }
Move after removal
  1. Initial state

    No operation at this step.

    Calculation state

    [
        "A",
        "B",
        "C",
        "D"
    ]
  2. Sequence succeeded

    Operation

    {
        "op": "move",
        "from": "/1",
        "path": "/3"
    }

    Calculation state

    [
        "A",
        "C",
        "D",
        "B"
    ]
Failed test and atomicity
  1. Initial state

    No operation at this step.

    Calculation state

    {
        "rev": 1,
        "label": "A"
    }
  2. Intermediate state, not committed

    Operation

    {
        "op": "replace",
        "path": "/label",
        "value": "B"
    }

    Calculation state

    {
        "rev": 1,
        "label": "B"
    }
  3. Failure: HTTP PATCH applies no changes

    Operation

    {
        "op": "test",
        "path": "/rev",
        "value": 2
    }

    Calculation state

    {
        "rev": 1,
        "label": "A"
    }
  4. Not executed after failure

    Operation

    {
        "op": "add",
        "path": "/extra",
        "value": true
    }

    Calculation state

    {
        "rev": 1,
        "label": "A"
    }
Missing parent
  1. Initial state

    No operation at this step.

    Calculation state

    {
        "a": []
    }
  2. Failure: HTTP PATCH applies no changes

    Operation

    {
        "op": "add",
        "path": "/missing/x",
        "value": 1
    }

    Calculation state

    {
        "a": []
    }
Escaped path and root
  1. Initial state

    No operation at this step.

    Calculation state

    {
        "a/b": {
            "~key": 1
        }
    }
  2. Intermediate state, not committed

    Operation

    {
        "op": "test",
        "path": "/a~1b/~0key",
        "value": 1
    }

    Calculation state

    {
        "a/b": {
            "~key": 1
        }
    }
  3. Sequence succeeded

    Operation

    {
        "op": "add",
        "path": "",
        "value": [
            "root"
        ]
    }

    Calculation state

    [
        "root"
    ]
add on an existing member
  1. Initial state

    No operation at this step.

    Calculation state

    {
        "x": 1
    }
  2. Sequence succeeded

    Operation

    {
        "op": "add",
        "path": "/x",
        "value": 2
    }

    Calculation state

    {
        "x": 2
    }

Implication for the record

This reader presents documented walkthroughs, not a universal JSON Patch executor. It contacts no server and predicts no response code. The atomicity shown belongs to HTTP PATCH; a local library must document its own error and mutation behavior.

Does PATCH alone specify how the body is processed?

No. The media type and resource contract are needed. Accept-Patch can advertise accepted formats. Neither the method name nor a local example proves that a remote API supports a format or permits the update.

Primary documentation

Documentation consulted on .

DOCUMENTED REFERENCE

SemVer: compare precedence without inventing identity

SemVer 2.0.0 defines syntax and precedence for software versions with a declared public API. Treating them as plain strings or decimal numbers gives incorrect results. A dataset date or tag name does not automatically adopt this contract.

SemVer: compare precedence without inventing identity
ReferenceValue or formWhat it identifies
MAJOR.MINOR.PATCH0 | [1-9][0-9]*The core contains three non-negative integers: MAJOR.MINOR.PATCH. Leading zeroes are forbidden except for 0 itself. The tag v1.2.3 may identify a release, but its full string is not a strict SemVer version.
1.9.0 / 1.10.0numericCore numbers are compared numerically, left to right. Thus 1.9.0 precedes 1.10.0. SemVer sets no numeric length limit: the tool compares digits without floating-point conversion.
-alpha.1pre-releaseWith equal cores, a prerelease precedes the normal version. Its dot-separated identifiers must be nonempty. A purely numeric identifier forbids leading zeroes; an identifier with a letter or hyphen follows a different case.
-2 / -alphanumeric < non-numericNumeric identifiers compare numerically; other identifiers use ASCII order. Numeric identifiers precede non-numeric ones. When all shared identifiers are equal, the prerelease with more identifiers has higher precedence.
+001ignored in precedenceMetadata after + never affects precedence. Unlike numeric prerelease identifiers, it may contain 001. Different strings can therefore have equal precedence without identifying equal files or interchangeable artifacts.

Compare two SemVer versions

Original teaching examples; no real release is evaluated.

Two strict SemVer 2.0.0 versions, at most 256 ASCII characters each. No spaces, v prefix or constraint operators. Numeric identifiers are compared exactly, without rounding.

Calculated locally in your browser; input is neither transmitted nor stored.

A precedes B in SemVer order.

1.9.0 < 1.10.0

First decisive criterion : MINOR

Version components
ComponentVersion AVersion B
MAJOR11
MINOR910
PATCH00
PrereleaseNoneNone
Build metadataNoneNone
Ten precedence comparisons
Ten precedence comparisons
Version A< / = / >Version BFirst decisive criterion
1.9.0<1.10.0MINOR
2.0.0>1.99.99MAJOR
1.0.9<1.0.10PATCH
1.2.3-rc.9<1.2.3Prerelease versus normal version
1.2.3-alpha.2<1.2.3-alpha.11Prerelease identifier
1.2.3-9<1.2.3-alphaPrerelease identifier
1.2.3-alpha<1.2.3-alpha.0Prerelease identifier count
1.2.3-Beta<1.2.3-alphaPrerelease identifier
1.2.3+linux.001=1.2.3+windows.002Metadata excluded from precedence
9007199254740992.0.0<9007199254740993.0.0MAJOR
Ten strict syntax cases
Ten strict syntax cases
InputDiagnosis
1.2.3Valid
0.0.0-0Valid
1.2.3+001Valid
1.2.3-01aValid
1.2.3--Valid
v1.2.3Invalid
1.02.3Invalid
1.2.3-01Invalid
1.2.3-alpha..1Invalid
1.2.3+Invalid

Original interpretation and examples based on SemVer 2.0.0, originally authored by Tom Preston-Werner; reference text under CC BY 3.0. SemVer 2.0.0 · CC BY 3.0

Continue checking

Implication for the record

The comparator checks syntax and precedence, not release notes, content, security or actual compatibility. It handles no dependency constraints, npm ranges, Python versions or Debian versions. The 256-character limit belongs to this form, not to SemVer.

Should equal-precedence versions be merged in an inventory?

Not automatically. Keep the full string and artifact reference: 1.2.3+linux and 1.2.3+windows have equal precedence, but metadata can distinguish builds. A digest and publication contract remain separate checks.

Primary documentation

Documentation consulted on .

DEEPER READING / PRACTICE

Preserve differences between dataset versions

A public file is replaced at the same location. The versioning should allow you to recover the state used in a calculation and understand the subsequent changes.

Updated on .

Priority checks

Keep file, date, source, hash and scheme. Compare lines, columns, definitions and values before attributing differences to an evolution of the measured phenomenon.

Interpretation pitfalls

A new download date does not necessarily mean a new content. A structure change can make a brutal comparison difficult to interpret.

Working record to retain

The version journal combines status, hash, change and effect on the analysis. The calculations explicitly cite the version used.