In-depth practical guides

Document a sample before publishing a measurement

Define population, selection, repetitions and exclusions so test results can be interpreted.

Updated :

Selected cards and measurement ledger. Illustrative scene.

AI-generated illustration

Which pages or observations does the result cover?

A rate from convenient cases describes those cases, not the whole site. NIST covers measurement plans; WCAG-EM structures conformance sampling. Their purposes and rules remain distinct.

  1. Define population

    Specify URLs, languages, dates, queries and devices. Name the unit: page, journey, response or session. Retain the initial list and version.

  2. Describe selection

    Separate deliberately critical cases from non-preferential draws. Cover templates, functions and complete processes. Document mechanism, order and replacements.

  3. Separate repetitions

    Ten repeats of one question are not ten different intentions. Keep case ID, repeat, time and context; describe dependence before estimating uncertainty.

  4. Publish exclusions

    Record technical failures and untested cases rather than silently removing them. Report subsets before aggregation; explain weighting and limits.

From observation to decision

The overall result depends on test composition

One hundred journeys were selected to cover desktop and mobile equally. Success means completing the journey under the same criterion; selection is not a representative traffic sample.

Fictional learning case: counts and dates illustrate the method and do not describe a real assignment.

Results by test context

Scale from 0 to 100%; 50 selected journeys per context.

  • Desktop: 20 / 5040%
  • Mobile: 35 / 5070%
Case reading grid
ObservationUseful checkNext action
Desktop: 20 successes out of 5020 / 50 = 40%.Report this subgroup’s result.
Mobile: 35 successes out of 5035 / 50 = 70%.Keep the contextual difference visible.
All tests: 55 of 100(20 + 35) / (50 + 50) = 55%.Describe tested cases without generalising to all visitors.
Your decision — What would hypothetical weights of 80% desktop and 20% mobile give?

Explained answer

0.8 × 40% + 0.2 × 70% = 46%. This illustrates sensitivity to weights, not a representative estimate validated by these tests alone.

Reference: NIST recommends defining goals and a sampling plan, then documenting exceptions. The journeys and results here are fictional. NIST — Data Collection for PPC

Situations and decisions

Typical situations for preparing a check. They do not describe completed assignments or actual observations.

Only easy pages

A fictional audit tests articles but also claims to cover search and forms. Add missing functions and journeys before generalising.

One repeated query

Ten illustrative responses to one query measure its variability, not coverage of ten independent user needs.

Worked exercise

Twelve checks do not cover twenty journeys

Fictional example: the values and documents teach the method and do not describe a real organisation.

Two defects were observed in the selected set. Also check which families are unrepresented.

If the table exceeds the screen, scroll horizontally; arrow keys work when the region is focused.

Starting information
ItemInventoryItems examined
Articles88
Profiles64
Forms40
Searches20

Your checks

  1. Calculate overall and per-family coverage.
  2. Name the entirely missing functions.
  3. Define the scope of the observed defect rate.
Read the answer

Answer and decision

Coverage is 12/20, or 60% of listed items: 100% of articles, about 66.67% of profiles and 0% of forms and searches. The 2/12 rate describes this selected batch, not automatically the whole site. Add missing functions before decisions concerning them.

If the case changes

A documented random sample raises a different statistical question; do not assign its properties to a convenience batch.

Record to retain

Keep inventory, selection, selection reason, defects and generalisation limits.

Record to retain

  • Population and initial list
  • Selection rule and units
  • Repeats, exclusions and context
  • Denominator, weights and limits

Official references

Related protocols

Checks before reaching a conclusion

  • Is population defined?
  • Are units distinct?
  • Are exclusions visible?
  • Does the conclusion respect scope?

A selection plan and observation log with explicit denominator.

Frequently asked questions

Should critical and random cases become one rate?

Not without explaining selection and weighting. Critical cases find issues; their share is not automatically their population frequency.

Can a panel be reused after redesign?

Yes for comparison, tracking changes. Extend it for new templates and declare removed cases.

How to choose representative cases of a site?

Identify the important components, content and actions and then select cases covering their variants. Documenting the criteria and exclusions allows you to understand the scope of control without claiming to have tested everything.

TOOLS / REGISTERS

Information to record

No universal sample size fits every audit. An interval cannot repair biased selection or ill-defined populations. Sampling cannot establish that every page was tested.

  • Access date and time zone
  • Exact URL
  • Sample / population
  • Selection unit and case identifier
  • Frame, mechanism and selection order
  • Repetitions and dependence between observations
  • Excluded and untested cases, with reasons
  • Metric / measurement
  • Limit / uncertainty
  • Decision / next action
  • Review owner
  • Next review

DOCUMENTED REFERENCE

Independent evaluation: splitting cases is not enough after answers influenced tuning

A score can be optimistic with correct formulas. scikit-learn distinguishes training, tuning and final evaluation. An agent has the same issue when instructions change after repeatedly inspecting a supposed generalization test.

TRAINVALIDATETESTfit → select → evaluate
Independent evaluation: splitting cases is not enough after answers influenced tuning
ReferenceValue or formWhat it identifies
HoldoutReserve cases before fittingReserve evaluation cases before adjustments; continual feedback into prompts, system or thresholds removes independence.
fit / transformFit only on training dataFit preprocessing on training data, then transform test data. Imputation, feature selection and normalization can leak information when fitted globally.
Threshold selectionValidation ≠ final testValidation selects thresholds or parameters; final test evaluates. Reporting the best of many thresholds on that same test also selects favorable noise.
GroupKFoldGroups separate within each splitKeep dependent group observations together. Near-document versions or repeated traces split across train/test may make the task artificially easy.
TimeSeriesSplitPast training → later testUse chronology when predicting future cases. TimeSeriesSplit gap counts observations, not a universal duration.
random_stateRepeatability ≠ representativenessA fixed seed repeats a split; it guarantees neither independence, coverage nor stability of a changing remote service.

Implication for the record

Document splitting unit, dependencies, exclusions, settings and versions. Split methods do not prove representativeness. Preserve dates and exact cases and reevaluate independently after relevant changes.

Can a corpus used to improve instructions remain a test set?

It now serves development/validation. Reserve new unseen cases to estimate generalization and state each set’s role.

Primary documentation

Documentation consulted on .

DOCUMENTED REFERENCE

Incomplete judgments: preserve unknowns and scope

Relevance is assessed for a particular information need. A partially judged collection does not make unjudged documents useless. This separate example distinguishes judgment coverage, observed precision and possible bounds.

Incomplete judgments: preserve unknowns and scope
ReferenceValue or formWhat it identifies
Judged / retrieved3 / 5 = 60%Separate original example: five results, two relevant, one nonrelevant and two unjudged. Judgment coverage is 3/5 = 60%.
P@5 bounds2 / 5 ≤ P@5 ≤ 4 / 5If both unknowns are nonrelevant, precision is 2/5 = 40%. If both are relevant, it is 4/5 = 80%. These are deterministic bounds, not a confidence interval.
P among judged2 / 3Precision among judged results is 2/3. It answers a different question and does not replace precision across all five positions. Removing unknowns also changes ranks.
Recall · collection?Without the collection’s complete relevant set, global recall cannot be inferred from these five results. Record the pool, instructions, assessor, collection version and date.
Separate example: five results and two missing judgments
RankJudgment status
1Relevant
2Unjudged
3Nonrelevant
4Relevant
5Unjudged

A missing judgment is not automatically assigned grade zero.

judged = 3/5 = 0.6
P@5 ∈ [2/5, 4/5] = [0.4, 0.8]
P(judged) = 2/3 ≈ 0.666667

Implication for the record

These five statuses are a teaching construction, not site observations. Pooling can leave relevant documents unjudged. Offline ranking metrics prove neither generated-answer factual accuracy nor satisfaction for every user.

What should a protocol include before comparing engines?

Judge the same queries and documents with the same instructions, retain unknowns and declare their treatment. Separate passage retrieval, answer faithfulness to passages and source validity; a ranking score does not cover all three.

Primary documentation

Documentation consulted on .

DOCUMENTED REFERENCE

Build the panel: 20 structured views, 2 drawn views, then missing process steps

WCAG-EM 2.0, the Group Note of 23 July 2026, describes a random addition equal to 10% of the structured sample set, excluding views already selected. The percentage applies to that set, not all pages of the website. The method also includes complete processes; a URL alone may miss successive form states.

Structured set20S01…S20
Fictional scenario: 20 distinct views cover the selected content families and functions. A view may cover several characteristics; that does not make it several views.
Random addition2R01, R02
Two distinct views are drawn outside the first set: 10% × 20 = 2. Retain the starting list, method and drawn identifiers. This calculation does not determine the size of the structured set.
Process addition3P01, P03, P04
Here R02 is step 2 of a four-view process. Steps 1, 3 and 4 are in neither set: they add three views, identified as P01, P03 and P04.
Scenario total25S ∪ R ∪ P
The sets are disjoint in this example: 20 + 2 + 3 = 25 views. Overlap would change the total. The count measures neither coverage of every possible state nor product conformance.

25 distinct views, three reasons for inclusion

Structured set
20
Random addition
2
Process addition
3
Each tile represents one view in this scenario, not an actual audited page of this site. Letters retain the reason for first inclusion: S, structured; R, random; P, process addition.

One address, four views to identify

In the fictional /demande service, navigation keeps the same URL. R02 shows the completed form at step 2. Testing that view alone documents neither the start nor the end of the process.

  1. P01 · open the form
  2. R02 · enter information
  3. P03 · review the summary
  4. P04 · display confirmation

If the table exceeds the screen, scroll it horizontally; with a keyboard, focus the region and use the arrow keys.

From a view count to a reproducible record
View or situationRecord to retainWhat remains to check
R02 · completed formURL /demande, step 2, language, version, fictional test data and required actions.Access the other steps; do not infer them from this screenshot.
Input errorSame URL, invalid field, displayed message and action triggering this state.Add this relevant branch: it is not yet included in the 25 views.
Another languageVariant identifier, language and reading direction.Observed French does not automatically cover Arabic; select its relevant interactions.
View cannot be openedPlanned identifier, attempt, blocker and any justified replacement.Keep the untested case visible; an easy replacement does not establish that the blocked view was checked.
What if the random selection reveals a new form type?

Revisit structured coverage and record additions, then compare findings from the two sets. Random selection also helps detect incomplete exploration. In this scenario, 25 is the current plan size, not a ceiling.

To reuse this case, keep one row per view with identifier, selection origin, context, actions, expected state and tested / untested status. Link steps to the same process; distinguish a missing observation from a successful check.

Implication for the record

These counts and identifiers are fictional. An evaluated sample alone does not establish WCAG conformance for the whole product. Critical branches also need attention; the four steps shown illustrate only the default path.

Primary documentation

Documentation consulted on .

DOCUMENTED REFERENCE

Read a real series: Internet use, 2020–2024

These observations compare percentage-point and relative changes in a published series. The three countries are selected for reading, not as a digital-performance ranking.

The indicator measures the share of the population using the Internet in the previous three months. It measures neither connection time, access quality nor AI-assistant use.

IT.NET.USER.ZS · Extracted · Latest update reported by the API .

Share of population · % · axis from 0 to 100Chart of three series; every value is available in the following table.025507510020202021202220232024FranceGermanySpain
Share of population · % · axis from 0 to 100 · FranceGermanySpain
Read a real series: Internet use, 2020–2024 · %
Country20202021202220232024
France84.7186.1085.3386.8488.65
Germany89.8191.4391.6392.4893.50
Spain93.2193.9094.4995.4595.76

France: from 84.71% in 2020 to 88.65% in 2024; 3.95 percentage points, or 4.66% relative to the starting value.

Table values are rounded to two decimals; calculations use the received values before that rounding. Examine revisions and collection differences before making a causal explanation.

Source: ITU, distributed by the World Bank. Extract and chart by Alexis Roux ZG. CC BY 4.0

Consult the official series · Definition and methodology

DEEPER READING / PRACTICE

Describe the sample before presenting a rate

A audit controls only a part of the body. The documentation must specify population, selection unit, selection mechanism and untested cases before the results are interpreted.

Updated on .

Priority checks

Keep a starting list, criteria, selection order and exclusions. Distinguish oriented sample, random selection, and pathways chosen to cover specific situations.

Interpretation pitfalls

A large number of tests does not correct an incomplete coverage. Repeated observations on the same page can give an impression of misleading diversity.

Working record to retain

The sample sheet indicates population, cases selected, exclusions and scope of the conclusion. The measures publish the actually observed denominator.