In-depth practical guides

Measure uncertainty in an AI citation rate

Separate observed rate, sample size and statistical uncertainty when interpreting an AI answer observatory.

Updated :

Evaluation of assistant answers and references on two screens.

AI-generated illustration

Is a citation rate precise enough to guide a decision?

A proportion without a denominator conceals its fragility. Four cited answers out of ten and forty out of one hundred have the same observed rate, but different precision under a binomial model. A Wilson interval explores that difference. It does not turn a selected question panel into a representative user sample.

  1. Define success and denominator

    Choose before collection what counts: a link, a citation supporting the claim or an accurate answer. Score each observation as binary under this rule. Document failures, exclusions and inaccessible responses without changing the rule after seeing results.

  2. Retain the protocol

    Fix language, interface, country, model or mode and period. Repeated prompts in one conversation may be dependent. Separate strata when conditions change; repeating one question does not create diverse needs.

  3. Read rate and interval

    The 95% Wilson interval here concerns independent binary observations with stable probability. It remains calculable with no or all successes. Confidence describes the method under repeated sampling, not a citation’s truth or panel representativeness.

  4. Decide with care

    Retain rate, counts, assumptions and decision threshold. If uncertainty blocks the decision, redesign collection before adding retries. A before/after comparison on the same questions is paired and needs a change analysis; two separate intervals are not that test.

Reading workshop

Two undecided labels do not disappear from the rate

The DEMO batch contains ten invented answers. Success means a citation supporting the claim, not merely a link being present.

Fictional data and records for learning: none of these results comes from a live service query.

40%, 50% or a classification range?

Four answers are labelled supported, four unsupported and two remain undecided after reading. The panel was selected for the exercise; independence and representativeness are not established.

If the table exceeds the screen, scroll horizontally; arrow keys work when the region is focused.

40%, 50% or a classification range?
ReadingCalculationWhat the number describesDecision
Annotation state4 supported; 4 unsupported; 2 undecidedTen received answers, not ten finalized binary labelsKeep undecided cases and their reasons.
Confirmed share of the batch4 / 10 = 40%Citation support already confirmed in this batchDo not call the two undecided cases established failures.
Rate among classified answers4 / 8 = 50%Rate conditional on a decided labelAlso report both exclusions and the total of ten.
Possible review outcomes4 / 10 to 6 / 10: 40–60%Classification range, not a confidence intervalResolve labels before using a binary model.

What can be reported without inventing statistical precision?

Open each proposition to compare its diagnosis. More checks may still be needed.

50% across ten answers, with no qualification

No. The denominator for 50% is eight. This hides the population change and both open labels.

40–60% is the 95% Wilson interval

No. These bounds come only from possible classification outcomes; no sampling probability was calculated.

Four supported citations out of ten, with two cases still undecided

Yes. Add the conditional rate if useful and describe the panel. Annotation uncertainty differs from sampling uncertainty.

Fictional annotation record

Batch: DEMO, ten selected answers
Rule: citation supports the claim
Decided: 4 supported and 4 unsupported
Undecided: 2, passages to review
Confirmed: 4/10; conditional: 4/8
Classification outcomes: 40–60%, no confidence level
Next: documented adjudication of both cases

This range quantifies neither future model behaviour nor panel bias. Do not automatically replace ten by eight in every calculation.

Documentation of the rules, separate from fictional examples

Compare other situations and exercises

Four out of ten

Four successes in ten give 40% and a 95% Wilson interval of about 16.8–68.7% under the stated assumptions. The rate alone hides this width and does not establish a universal ranking.

Zero out of ten

The observed rate is zero, but the upper bound is about 27.8%. No citations in this sample does not establish that citation is impossible.

A high rate may still miss the decision threshold

Eighteen successes in twenty give 90%. Under the stated binomial assumptions, the 95% Wilson interval is about 69.9–97.2%. If a decision requires more than 80%, the lower bound does not confirm it. Report counts and the decision rule.

Twenty reruns are not twenty situations

Fictional case: one question is rerun twenty times in one conversation, while another panel contains twenty different needs. Equal raw rates do not make diversity or dependence comparable. Keep question and session identifiers and describe groups before calculating an overall metric.

From observation to decision

The same observed rate, very different precision

Two fictional collections give 40%: 4 successes in 10 and 40 in 100. Calculation assumes independent binary observations with stable probability. Intervals describe that assumption without establishing question representativeness.

Fictional learning case: counts and dates illustrate the method and do not describe a real assignment.

Read the width as well as the point

Left to right: 0 to 100%. Band: 95% Wilson interval; vertical marker: observed rate of 40%.

  • 4 / 1040% · 16.8–68.7%
  • 40 / 10040% · 30.9–49.8%
Case reading grid
ObservationUseful checkNext action
4 successes in 10Rate: 40%; 95% Wilson interval: 16.8–68.7%.A width of about 51.9 percentage points leaves substantial uncertainty.
40 successes in 100Rate: 40%; 95% Wilson interval: 30.9–49.8%.Width is about 18.9 percentage points; the protocol still needs review.
Same proportion, different basesCount changes precision under the model.Report successes, total, interval and assumptions together.
Your decision — Do a hundred repeats mean a hundred situations?

Explained answer

No. Row count establishes neither independence nor diversity. Keep question, session and conditions; examine dependent response groups. If the protocol cannot support probabilistic inference, describe the panel results as descriptive.

Method: two-sided 95% Wilson interval, z ≈ 1.96. Counts are fictional; bounds are rounded to a tenth of a percentage point. NIST — Confidence intervals for proportions

Worked exercise

Four unreadable answers are not four missing citations

Fictional example: the values and documents teach the method and do not describe a real organisation.

Panel categories are disjoint. An unreadable answer is not an observed absence of citation.

If the table exceeds the screen, scroll horizontally; arrow keys work when the region is focused.

Starting information
ObservationCount
Observed citation12
Readable without citation4
Unable to examine4

Your checks

  1. Separate reading failure from observed absence.
  2. Calculate the rate on readable answers.
  3. Give descriptive bounds for the whole panel.
Read the answer

Answer and decision

Among 16 readable answers the rate is 12/16 = 75%. Across 20, citations range from 12 to 16, or 60–80%, depending on the unknowns. These are missing-data bounds, not a Wilson confidence interval. Processing does not turn unknown answers into observations.

If the case changes

After recovering four answers, replace unknowns with actual states and publish the revised summary version.

Record to retain

Keep counts by state, denominators, unreadability reason and exclusion rule.

Record to retain

  • Success definition, question panel, exclusions and collection context.
  • Success count k, observation count n and series date.
  • Wilson method, 95% level, assumptions and decision threshold.
  • Preselected threshold, any conditional rate, sessions and access failures separately recorded.

Official references

Related protocols

Checks before reaching a conclusion

  • Was success defined before counting?
  • Are observations independent and the protocol stable?
  • Is the result separate from assessment of citation accuracy?

Frequently asked questions

Does 95% mean 95% of answers will be cited?

No. The rate is k/n. The 95% level qualifies the interval method under the stated assumptions.

Do overlapping intervals prove no difference?

No. Overlap is not a difference test, and paired observations require appropriate treatment.

How should a technically inaccessible answer be counted?

Set the rule before collection. A rate conditional on readable answers needs the access-failure count and total attempts. Do not automatically classify every inaccessible answer as an absent citation or silently exclude it.

How to deal with a difficult to classify case?

Previne an uncertain category and describe the reason. A second reading may clarify the judgment; it should not delete the limits or impose a certainty that the case does not allow.

MEASUREMENT / VISIBILITY SOURCES

Impression, citation, visit: three different observations

Appearing in an AI answer and receiving a website visit happen at different stages. Choose the signal that answers your question before comparing periods.

  1. Be accessible

    Crawling may make a page available for search.

  2. Be shown or cited

    A link can appear in an answer without receiving a click.

  3. Receive a visit

    The site records an arrival within the limits of its measurement setup.

Which report answers which question?
Observation sourceWhat is measuredInterpretation limit
Google · Generative AI in Search Console
Official documentation
Link impressions in AI Overviews and AI Mode, with dimensions including page, country and device.Days use Pacific Time (PT). Property totals can differ from the sum of page rows because of aggregation. If the report does not appear, Google says the site usually has too few impressions in these features.
Bing Webmaster Tools · AI Performance
Official documentation · Bing announcement
Visible citations and cited pages in supported AI experiences; grounding queries are grouped.A citation is neither a click, a ranking nor a measure of page importance. Grouped queries do not reproduce individual prompts. The queries shown reflect a sample of citation activity.
Website measurementArrivals and actions recorded under your measurement configuration.An attributed visit does not represent every impression. Missing referrers or blocked measurement limit attribution.
Documented question panelCitation presence and fidelity across a set of observed answers.The result depends on questions, mode, language and date. It does not describe all users.

Google AI report impressions are included in the Search Console Web scope: do not add them to the Web total. Compare visibility, citations and visits separately.

Compare the scope of two observations

Documentation accessed on .

DOCUMENTED REFERENCE

Spread and uncertainty: distinguish SD from standard error

Spread describes observations; estimator uncertainty describes sampling variability within a model. Sharing units does not make them interchangeable. R documents its SD denominator; NIST explains assumptions for mean intervals.

Spread and uncertainty: distinguish SD from standard error
ReferenceValue or formWhat it identifies
Sample varianceΣ(xᵢ − mean)² / (n − 1)Sample variance here divides by n − 1 for retained observations. Division by n differs; record the denominator.
Sample SD√varianceSquare root of variance, back in original units. Variance has squared units. Large SD alone does not prove collection error.
IQRQ3 − Q1Q3 minus Q1 describes central spread and depends on quartile convention; preserve algorithm.
Rangemax − minMaximum minus minimum responds strongly to extremes; not a confidence interval or future coverage probability.
SE(mean)s / √n · assumptions requiredSimple s/√n requires an appropriate framework, including independence. Not sufficient for every weighted survey, time series or repeated observation.
n < 2R sd(): NADocumented R sd() returns NA for zero or one observation. Missing is not automatically zero SD.

Implication for the record

Mean intervals require model, level and method, not just plus/minus SD. AI citations or repeated outputs may be dependent; more rows do not ensure independent-observation uncertainty reduction.

Can SD and SE bars be compared?

Only after identifying their meaning. SD describes observation spread, SE mean estimation within a model. Shorter bars alone do not prove a less dispersed population.

Primary documentation

Documentation consulted on .

DOCUMENTED REFERENCE

Observed proportion: report counts and a Wilson interval

Five positive cases out of twenty give 25%, as do 250 out of 1,000. Under a suitable binomial model, the estimates do not have equal precision. This explorer separates the observed rate, confidence level and calculated limits.

Observed proportion: report counts and a Wilson interval
ReferenceValue or formWhat it identifies
Observed proportionx / nx counts positive cases under a fixed criterion; n counts all assessable cases. Missing or undecidable responses are not negatives: document exclusions separately.
Wilson scoretwo-sidedWilson is a two-sided score interval. Its center generally differs from x/n and its limits stay between 0 and 1. It is neither the Wald interval nor the exact Clopper–Pearson binomial interval.
Confidence level90% · 95% · 99%Levels are 90%, 95% and 99%, using the corresponding normal quantile. Confidence describes repeated-sampling coverage of the procedure, not a posterior probability for this one interval.
x = 0upper limit > 0When x = 0 and n > 0, the upper limit remains positive. No observed events do not prove impossibility. No interval is calculated when n = 0.
Sampling modelindependent · common probabilityThe model assumes independent trials with a common probability. Repeated answers to one query, clustered cases and convenience samples do not automatically meet those conditions.

Explore a proportion

Teaching exercise: no observations of a real service. Positive and negative refer only to the declared criterion.

Model assumptions: independent trials with a common probability. Selected samples and dependent repetitions do not become representative through this interval.

Enter integers with 1 ≤ n ≤ 1,000,000 and 0 ≤ x ≤ n.

Calculations stay in your browser; no inputs are transmitted and no file is created.

Results calculated · x = 5 · n = 20 · 95%

Observed rate (%)
25
Lower limit (%)
11.18617
Upper limit (%)
46.870088
Width (percentage points)
35.683918
Wilson interval and observed rateThe point shows x/n; the line is the two-sided Wilson interval. The axis stays at 0–100%. Values are also listed above.0%25%50%75%100%Wilson · 95%
The point shows x/n; the line is the two-sided Wilson interval. The axis stays at 0–100%. Values are also listed above.

Formula and calculation convention

z = {"90":1.6448536269514715,"95":1.9599639845400536,"99":2.5758293035489}
p = x/n; D = 1 + z²/n
center = (p + z²/(2n)) / D
half = z × sqrt(p(1−p)/n + z²/(4n²)) / D
Wilson = [center − half, center + half]

Very small p-values use scientific notation instead of displaying zero. Other results are rounded to six decimals.

Calculated 95% examples: this table keeps its level when the selector changes.
x / nObserved rate (%)Lower limit (%)Upper limit (%)Width (percentage points)
0 / 50043.44824643.448246
1 / 5203.62241162.44653758.824126
5 / 510056.55175410043.448246
5 / 202511.1861746.87008835.683918
25 / 1002517.54521134.30446416.759252
250 / 10002522.4153127.7760285.360718
0 / 100003.699353.69935
100 / 10010096.300651003.69935

All examples are constructed to explain calculations, not commercial performance data.

Confidence is not the probability that a claim is true. A p-value does not measure that probability either.

Implication for the record

Original examples, not measurements of a real service. Wilson coverage is approximate, not guaranteed to be 95% at every sample size. It corrects neither selection bias, dependence nor annotation errors and does not predict future engine changes.

What should accompany a citation rate?

Report x and n, the positive criterion, exclusions, queries and selection, versions, dates, independent unit and interval method. Citation presence does not establish citation correctness; evaluate that separately.

Primary documentation

Documentation consulted on .

DEEPER READING / PRACTICE

State what the results support

Two series of answers show different quotation rates. The issue is to determine whether the gap describes a reproducible variation or simply the composition of the questions tested.

Updated on .

Priority checks

Examine staff, repetitions and the dependencies between observations. Separate the results by category when the questions relate to different intentions or levels of difficulty.

Interpretation pitfalls

A precise percentage can give a false impression of solidity. A small sample or oriented selection remains limited, even accompanied by decimals and graphics.

Working record to retain

An uncertainty note shows the number of cases, the classification rule and the variations observed. It specifies the defendable comparisons and the additional observations required.

EXAMPLE / READING THE NUMBERS

Presence does not always mean accurate representation

Fictional data illustrating the method; not client results or measurements of this website.

Presence does not always mean accurate representation
ObservationCountInterpretation
Panel questions40Fixed comparison base.
Answers mentioning the brand1212 / 40 = 30% of the panel.
Answers describing the offer correctly88 / 40 = 20% of the panel; 8 / 12 ≈ 66.7% of mentions.
Show the interpretation limit

What these numbers do not prove

These observations measure neither market share nor sales. A mention may be inaccurate. Report the panel, date, language and repetitions alongside the result.