Only easy pages
A fictional audit tests articles but also claims to cover search and forms. Add missing functions and journeys before generalising.
In-depth practical guides
Define population, selection, repetitions and exclusions so test results can be interpreted.
Updated :

AI-generated illustration
A rate from convenient cases describes those cases, not the whole site. NIST covers measurement plans; WCAG-EM structures conformance sampling. Their purposes and rules remain distinct.
Specify URLs, languages, dates, queries and devices. Name the unit: page, journey, response or session. Retain the initial list and version.
Separate deliberately critical cases from non-preferential draws. Cover templates, functions and complete processes. Document mechanism, order and replacements.
Ten repeats of one question are not ten different intentions. Keep case ID, repeat, time and context; describe dependence before estimating uncertainty.
Record technical failures and untested cases rather than silently removing them. Report subsets before aggregation; explain weighting and limits.
From observation to decision
One hundred journeys were selected to cover desktop and mobile equally. Success means completing the journey under the same criterion; selection is not a representative traffic sample.
Fictional learning case: counts and dates illustrate the method and do not describe a real assignment.
Scale from 0 to 100%; 50 selected journeys per context.
| Observation | Useful check | Next action |
|---|---|---|
| Desktop: 20 successes out of 50 | 20 / 50 = 40%. | Report this subgroup’s result. |
| Mobile: 35 successes out of 50 | 35 / 50 = 70%. | Keep the contextual difference visible. |
| All tests: 55 of 100 | (20 + 35) / (50 + 50) = 55%. | Describe tested cases without generalising to all visitors. |
0.8 × 40% + 0.2 × 70% = 46%. This illustrates sensitivity to weights, not a representative estimate validated by these tests alone.
Reference: NIST recommends defining goals and a sampling plan, then documenting exceptions. The journeys and results here are fictional. NIST — Data Collection for PPC
Typical situations for preparing a check. They do not describe completed assignments or actual observations.
A fictional audit tests articles but also claims to cover search and forms. Add missing functions and journeys before generalising.
Ten illustrative responses to one query measure its variability, not coverage of ten independent user needs.
Worked exercise
Fictional example: the values and documents teach the method and do not describe a real organisation.
Two defects were observed in the selected set. Also check which families are unrepresented.
If the table exceeds the screen, scroll horizontally; arrow keys work when the region is focused.
| Item | Inventory | Items examined |
|---|---|---|
| Articles | 8 | 8 |
| Profiles | 6 | 4 |
| Forms | 4 | 0 |
| Searches | 2 | 0 |
Coverage is 12/20, or 60% of listed items: 100% of articles, about 66.67% of profiles and 0% of forms and searches. The 2/12 rate describes this selected batch, not automatically the whole site. Add missing functions before decisions concerning them.
A documented random sample raises a different statistical question; do not assign its properties to a convenience batch.
Keep inventory, selection, selection reason, defects and generalisation limits.
A selection plan and observation log with explicit denominator.
Not without explaining selection and weighting. Critical cases find issues; their share is not automatically their population frequency.
Yes for comparison, tracking changes. Extend it for new templates and declare removed cases.
Identify the important components, content and actions and then select cases covering their variants. Documenting the criteria and exclusions allows you to understand the scope of control without claiming to have tested everything.
TOOLS / REGISTERS
No universal sample size fits every audit. An interval cannot repair biased selection or ill-defined populations. Sampling cannot establish that every page was tested.
DOCUMENTED REFERENCE
A score can be optimistic with correct formulas. scikit-learn distinguishes training, tuning and final evaluation. An agent has the same issue when instructions change after repeatedly inspecting a supposed generalization test.
| Reference | Value or form | What it identifies |
|---|---|---|
| Holdout | Reserve cases before fitting | Reserve evaluation cases before adjustments; continual feedback into prompts, system or thresholds removes independence. |
| fit / transform | Fit only on training data | Fit preprocessing on training data, then transform test data. Imputation, feature selection and normalization can leak information when fitted globally. |
| Threshold selection | Validation ≠ final test | Validation selects thresholds or parameters; final test evaluates. Reporting the best of many thresholds on that same test also selects favorable noise. |
| GroupKFold | Groups separate within each split | Keep dependent group observations together. Near-document versions or repeated traces split across train/test may make the task artificially easy. |
| TimeSeriesSplit | Past training → later test | Use chronology when predicting future cases. TimeSeriesSplit gap counts observations, not a universal duration. |
| random_state | Repeatability ≠ representativeness | A fixed seed repeats a split; it guarantees neither independence, coverage nor stability of a changing remote service. |
Document splitting unit, dependencies, exclusions, settings and versions. Split methods do not prove representativeness. Preserve dates and exact cases and reevaluate independently after relevant changes.
It now serves development/validation. Reserve new unseen cases to estimate generalization and state each set’s role.
Documentation consulted on .
DOCUMENTED REFERENCE
Relevance is assessed for a particular information need. A partially judged collection does not make unjudged documents useless. This separate example distinguishes judgment coverage, observed precision and possible bounds.
| Reference | Value or form | What it identifies |
|---|---|---|
| Judged / retrieved | 3 / 5 = 60% | Separate original example: five results, two relevant, one nonrelevant and two unjudged. Judgment coverage is 3/5 = 60%. |
| P@5 bounds | 2 / 5 ≤ P@5 ≤ 4 / 5 | If both unknowns are nonrelevant, precision is 2/5 = 40%. If both are relevant, it is 4/5 = 80%. These are deterministic bounds, not a confidence interval. |
| P among judged | 2 / 3 | Precision among judged results is 2/3. It answers a different question and does not replace precision across all five positions. Removing unknowns also changes ranks. |
| Recall · collection | ? | Without the collection’s complete relevant set, global recall cannot be inferred from these five results. Record the pool, instructions, assessor, collection version and date. |
| Rank | Judgment status |
|---|---|
| 1 | Relevant |
| 2 | Unjudged |
| 3 | Nonrelevant |
| 4 | Relevant |
| 5 | Unjudged |
A missing judgment is not automatically assigned grade zero.
judged = 3/5 = 0.6 P@5 ∈ [2/5, 4/5] = [0.4, 0.8] P(judged) = 2/3 ≈ 0.666667
These five statuses are a teaching construction, not site observations. Pooling can leave relevant documents unjudged. Offline ranking metrics prove neither generated-answer factual accuracy nor satisfaction for every user.
Judge the same queries and documents with the same instructions, retain unknowns and declare their treatment. Separate passage retrieval, answer faithfulness to passages and source validity; a ranking score does not cover all three.
Documentation consulted on .
DOCUMENTED REFERENCE
WCAG-EM 2.0, the Group Note of 23 July 2026, describes a random addition equal to 10% of the structured sample set, excluding views already selected. The percentage applies to that set, not all pages of the website. The method also includes complete processes; a URL alone may miss successive form states.
In the fictional /demande service, navigation keeps the same URL. R02 shows the completed form at step 2. Testing that view alone documents neither the start nor the end of the process.
If the table exceeds the screen, scroll it horizontally; with a keyboard, focus the region and use the arrow keys.
| View or situation | Record to retain | What remains to check |
|---|---|---|
| R02 · completed form | URL /demande, step 2, language, version, fictional test data and required actions. | Access the other steps; do not infer them from this screenshot. |
| Input error | Same URL, invalid field, displayed message and action triggering this state. | Add this relevant branch: it is not yet included in the 25 views. |
| Another language | Variant identifier, language and reading direction. | Observed French does not automatically cover Arabic; select its relevant interactions. |
| View cannot be opened | Planned identifier, attempt, blocker and any justified replacement. | Keep the untested case visible; an easy replacement does not establish that the blocked view was checked. |
Revisit structured coverage and record additions, then compare findings from the two sets. Random selection also helps detect incomplete exploration. In this scenario, 25 is the current plan size, not a ceiling.
To reuse this case, keep one row per view with identifier, selection origin, context, actions, expected state and tested / untested status. Link steps to the same process; distinguish a missing observation from a successful check.
These counts and identifiers are fictional. An evaluated sample alone does not establish WCAG conformance for the whole product. Critical branches also need attention; the four steps shown illustrate only the default path.
Documentation consulted on .
DOCUMENTED REFERENCE
These observations compare percentage-point and relative changes in a published series. The three countries are selected for reading, not as a digital-performance ranking.
The indicator measures the share of the population using the Internet in the previous three months. It measures neither connection time, access quality nor AI-assistant use.
IT.NET.USER.ZS · Extracted · Latest update reported by the API .
| Country | 2020 | 2021 | 2022 | 2023 | 2024 |
|---|---|---|---|---|---|
| France | 84.71 | 86.10 | 85.33 | 86.84 | 88.65 |
| Germany | 89.81 | 91.43 | 91.63 | 92.48 | 93.50 |
| Spain | 93.21 | 93.90 | 94.49 | 95.45 | 95.76 |
France: from 84.71% in 2020 to 88.65% in 2024; 3.95 percentage points, or 4.66% relative to the starting value.
Table values are rounded to two decimals; calculations use the received values before that rounding. Examine revisions and collection differences before making a causal explanation.
Source: ITU, distributed by the World Bank. Extract and chart by Alexis Roux ZG. CC BY 4.0
DEEPER READING / PRACTICE
A audit controls only a part of the body. The documentation must specify population, selection unit, selection mechanism and untested cases before the results are interpreted.
Updated on .
Keep a starting list, criteria, selection order and exclusions. Distinguish oriented sample, random selection, and pathways chosen to cover specific situations.
A large number of tests does not correct an incomplete coverage. Repeated observations on the same page can give an impression of misleading diversity.
The sample sheet indicates population, cases selected, exclusions and scope of the conclusion. The measures publish the actually observed denominator.