In-depth practical guides

Test internal search relevance

Build representative queries, check results by language and improve journeys without confusing the existence of search with successful discovery.

Updated :

Interface checks on a computer, tablet and phone.

AI-generated illustration

Can users find the content that answers their need?

Internal search serves different intentions: finding a known profile, exploring a topic or choosing a tool. A returned result does not prove useful content is visible. Connect each query to a task and check title, excerpt, destination and language. A documented set of meaningful questions is more useful than a score based on purposeless queries.

  1. Build a task set

    Select important content in each main family: guide, profile, category, tool and dossier. Describe the need and pages that actually answer it. Include a need with no known answer to test empty results.

  2. Vary meaningful wording

    Test exact title, everyday expression, specialist term and shorter wording. Include accents, apostrophes, non-Latin characters and abbreviations actually used. Variants should represent plausible usage rather than merely mirror editorial vocabulary.

  3. Check every language separately

    Search from all six interfaces. Check filter labels, excerpts, messages and destination language. For a localised corpus, results should not silently lead to another language version.

  4. Evaluate result and destination

    Record the expected page’s position and whether its title is recognisable. Open it and confirm useful content still exists. Stale excerpts, deleted profiles and overly broad categories can mislead despite seemingly good rankings.

  5. Check journey states

    Test empty query, no results, active filter, pagination and back navigation. Confirm query and language retention. Use the keyboard and check that the form’s label describes the action and messages remain understandable.

  6. Fix the cause and repeat

    Distinguish missing content, stale index, vocabulary differences, wrong links and unsuitable ranking. Apply the corresponding fix. Rebuild the index and repeat tasks to verify improvement and catch regressions elsewhere.

  7. Define several acceptable answers

    A broad intent may have several useful pages. Before testing, separate directly useful answers, acceptable supporting material and irrelevant results. Record whether the reader wants an explanation, a profile or a downloadable template. This evaluates ranking without artificially requiring one URL for every wording.

Reading workshop

Read a ranking with a declared cutoff

A fictional five-result set calculates precision, recall and reciprocal rank without changing denominators midway.

Fictional data and records for learning: none of these results comes from a live service query.

Query DEMO-Q03 and its relevance judgments

The ranking is A, B, C, D, E. The relevance reference contains C, E and F. Only the first three results are inspected: k=3 and m=3 results actually retained.

If the table exceeds the screen, scroll horizontally; arrow keys work when the region is focused.

Query DEMO-Q03 and its relevance judgments
Calculation elementScenario valueInterpretationCheck to retain
Results through the cutoffA, B, C; m=3E exists in the ranking but remains outside the cutoff.Fix the cutoff before calculation and retain original order.
Relevant reference setC, E, F; three answersF is absent from the returned results.Retain judgment rationales and the corpus version.
Intersection at k=3C only; one relevant result retrievedPrecision = 1/3; recall = 1/3.Here precision divides by m, the number actually retained through k; declare this convention.
First relevant resultC at rank 3; reciprocal rank RR = 1/3This is one query, not an MRR average.Do not present this isolated case as overall search-engine performance.

Evaluate your ranking against your relevance reference

One query, exact ASCII identifiers (letters, digits, dot, hyphen, colon or underscore), at most 32 characters. One identifier per line; surrounding whitespace and blank lines ignored. Up to 20 ranked results and 100 relevant references, without duplicates. Cutoff k from 1 to 20.

Results for the supplied set

Requested cutoff k
3
Examined results m
3
Relevant results found
1
Relevant reference documents
3
Precision over m results (%)
33.33
Recall within the reference (%)
33.33
Rank of the first relevant result
3
Reciprocal rank, one query
0.3333

m is the number actually supplied up to k. Precision = relevant results found / m; recall = relevant results found / reference size. A zero denominator leaves the rate undefined. Reciprocal rank is 1/first relevant rank, or zero without a hit.

Positions actually examined
RankExact identifierSupplied judgment
1AAbsent from the reference
2BAbsent from the reference
3CRelevant in the reference

Supplied judgments define this calculation; documents outside the reference are treated as nonrelevant. An incomplete reference does not give global recall. No search engine is queried. This reciprocal rank is not mean MRR; when m is smaller than k, precision uses m rather than a fixed-k denominator.

Which summary respects both cutoff and judgments?

Open each proposition to compare its diagnosis. More checks may still be needed.

Precision 2/5 because C and E are relevant.

That uses all five results although the declared cutoff is three. It answers a different question.

Recall 1/5 because the ranking has five items.

Wrong denominator. The relevant reference has three items, and only one was retrieved within the cutoff.

At k=3: precision 1/3, recall 1/3 and RR 1/3.

Correct for this case with m=3. The fractions coincide for different reasons; they are not interchangeable.

Fictional calculation record

Query: DEMO-Q03
Ranking: A, B, C, D, E
Relevant reference: C, E, F
Cutoff: k=3; actually retained: m=3
Intersection: C; count: 1
Precision: 1/3; recall: 1/3
First relevant: rank 3; RR: 1/3

A to F are invented identifiers. Judgments are assumptions of the exercise, not results of a real website evaluation.

Documentation of the rules, separate from fictional examples

Compare other situations and exercises

The page exists but is not returned

Search its title and inspect the localised index. If absent, fix inclusion or extraction. If present, examine indexed terms before creating duplicate content.

A filter creates a dead end

Check that it limits the intended type and remains visible when there are no results. Offer removal or useful rewording, not invented results to fill the screen.

Opening a result changes language

Compare its URL with the equivalent content. Fix parameters or routing and test pagination and return navigation. A translated title is insufficient if the destination resets language.

A guide and its template share a title

Illustrative case: identical titles open either a method or a blank document. Test understanding the method and obtaining the template separately. Rank for the task and show the result type; a technically correct link can still open the wrong kind of answer.

A short acronym creates incidental matches

Test example: API should not match a page solely because the word capital contains those letters. Check short words as units, then acronyms in titles and body text. Keep one relevant case and one excluded case so a change does not inflate recall artificially.

Worked exercise

Measure search against the completed task

Fictional example: the values and documents teach the method and do not describe a real organisation.

Task: verify a many-to-one join. The relevance judgments below are exercise inputs.

If the table exceeds the screen, scroll horizontally; arrow keys work when the region is focused.

Starting information
RankFictional resultTask supported
1General articleNo
2Join checking guideYes
3Entity profileNo
4Worked join caseYes
5Cardinality checksYes

Your checks

  1. Count results that actually satisfy the task.
  2. Calculate precision across the five results.
  3. Record the first useful rank and its reciprocal.
Read the answer

Answer and decision

Precision at five is 3/5, or 60%. The first useful result is at rank 2; reciprocal rank is 1/2. These measures distinguish list noise from access to the first useful answer.

If the case changes

A result mentioning joins without explaining the check does not satisfy this task. Reassess it if the content changes.

Record to retain

Keep query, task, five destinations, relevance judgments and check date.

DOCUMENTED REFERENCE

Top results: separate precision, recall and first success

Two rankings may retrieve the same number of relevant documents while placing them at very different positions. This exercise separates quality within the first k results, coverage of the collection and the wait for the first useful result.

Top results: separate precision, recall and first success
ReferenceValue or formWhat it identifies
CorpusA–H · R = 4In this original exercise, all eight documents A–H are judged for one query. A grade above zero is relevant: B, C, D and F, giving R = 4.
P@k / Recall@khits / k · hits / RPrecision divides relevant retrieved results by k. Recall uses all four relevant documents in the collection, including those not returned.
RR@k1 / first relevant rankRR@k is the reciprocal rank of the first relevant result within the window, or zero if none appears. This is a per-query measure, not a mean MRR.
AP@k · R denominatorΣ P@i × rel(i) / RThe convention here is AP@k = the sum of precision at relevant ranks through k, divided by R. Unretrieved relevant documents contribute zero. Some tools use a different denominator.

Explore the ranking

Original exercise: fixed grades for one query, with no personal data or request sent to an engine. A = 0, B = 3, C = 1, D = 2, E = 0, F = 3, G = 0, H = 0. IDs do not represent real documents.

Enter 1–8 distinct IDs A–H, separated by commas. Omitted documents remain in the reference collection.

Calculations stay in your browser. The two explorers are independent.

Initial ranking A,C,E,B; k = 4. The ideal reference also includes D and F, absent from the returned list.

Results for one query

Precision P@k
0.5
Recall@k
0.5
First success RR@k
0.5
AP@k, denominator R = 4
0.25
P@k = hits/k; Recall@k = hits/4; RR@k = 1/first relevant rank (otherwise 0); AP@k = Σ[rel(i) × P@i]/4
Top-rank detail
RankDocumentGradeCumulative relevant documentsPrecision at rankAP contribution
1A0000
2C110.50.125
3E010.3333330
4B320.50.125
Fully judged reference collection
DocumentGradeRelevant if grade > 0
A0No
B3Yes
C1Yes
D2Yes
E0No
F3Yes
G0No
H0No

Implication for the record

Teaching example, not a measurement of a real engine. Document IDs must be unique and fully ordered; there is no implicit tie handling. Results are rounded to six decimals; calculations use unrounded values.

Why can equal precision produce different reciprocal ranks?

A,C,E,B and B,A,E,C each contain two relevant documents in four results. Both precision and recall are 0.5, but the first success is at rank 2 or rank 1, so RR@4 is 0.5 or 1.

Primary documentation

Documentation consulted on .

Record to retain

  • Task, exact query, language, filters and justified expected pages.
  • Observed rank, excerpt, opened URL and useful content.
  • Empty state, keyboard journey and pagination behaviour.
  • Index version, cause, correction and retest outcome.
  • Intent, acceptable answer types and excluded examples with reasons.

Official references

Related protocols

Checks before reaching a conclusion

  • Does the query set include a no-result search, a homonym and a phrase found only in body text?
  • Do result links open the correct section and preserve language, parameters and useful anchors?
  • Are expected results and observed failures retained with the corpus version used for acceptance testing?
  • Does ranking distinguish explanations and templates while keeping other useful answers discoverable?

Frequently asked questions

How many queries should be tested?

Cover main families and intentions, then actual problem cases. Add queries revealing distinct gaps. Count alone does not establish representativeness.

Must every query return results?

No. Honest empty results with visible filters and useful suggestions beat irrelevant matches. Then examine whether the need warrants valuable new content.

How should improvement be measured?

Repeat the same tasks and compare success, useful-page rank and required steps. Keep environment and corpus version; more results can hide lower relevance.

Should the index retain everything?

Index useful searchable content and parameters needed for identity. Exclude errors, stale content and valueless technical states. Check that corpus updates trigger the intended rebuild.

How to choose the test requests?

Use the public questions, the operations described by the guides and some useful synonyms. Adding cases without expected response also allows you to check the understanding of the empty results.

DEEPER READING / PRACTICE

Check that the search finds useful content.

A page exists, but natural requests do not make it appear. The control must test the reader’s words, variants and filters with explicit expectations.

Updated on .

Priority checks

Create navigation, subject and method requests. Check the presence of the expected result, relevance of the description, destination and behavior of the filters.

Interpretation pitfalls

A search may seem effective on accurate titles but fail on the usual formulations. An old index may ignore a recently added content.

Working record to retain

The search paper combines query, intention, expected result and observed result. Corpus updates trigger a verification of new useful formulations.