Four out of ten
Four successes in ten give 40% and a 95% Wilson interval of about 16.8–68.7% under the stated assumptions. The rate alone hides this width and does not establish a universal ranking.
In-depth practical guides
Separate observed rate, sample size and statistical uncertainty when interpreting an AI answer observatory.
Updated :

AI-generated illustration
A proportion without a denominator conceals its fragility. Four cited answers out of ten and forty out of one hundred have the same observed rate, but different precision under a binomial model. A Wilson interval explores that difference. It does not turn a selected question panel into a representative user sample.
Choose before collection what counts: a link, a citation supporting the claim or an accurate answer. Score each observation as binary under this rule. Document failures, exclusions and inaccessible responses without changing the rule after seeing results.
Fix language, interface, country, model or mode and period. Repeated prompts in one conversation may be dependent. Separate strata when conditions change; repeating one question does not create diverse needs.
The 95% Wilson interval here concerns independent binary observations with stable probability. It remains calculable with no or all successes. Confidence describes the method under repeated sampling, not a citation’s truth or panel representativeness.
Retain rate, counts, assumptions and decision threshold. If uncertainty blocks the decision, redesign collection before adding retries. A before/after comparison on the same questions is paired and needs a change analysis; two separate intervals are not that test.
Reading workshop
The DEMO batch contains ten invented answers. Success means a citation supporting the claim, not merely a link being present.
Fictional data and records for learning: none of these results comes from a live service query.
Four answers are labelled supported, four unsupported and two remain undecided after reading. The panel was selected for the exercise; independence and representativeness are not established.
If the table exceeds the screen, scroll horizontally; arrow keys work when the region is focused.
| Reading | Calculation | What the number describes | Decision |
|---|---|---|---|
| Annotation state | 4 supported; 4 unsupported; 2 undecided | Ten received answers, not ten finalized binary labels | Keep undecided cases and their reasons. |
| Confirmed share of the batch | 4 / 10 = 40% | Citation support already confirmed in this batch | Do not call the two undecided cases established failures. |
| Rate among classified answers | 4 / 8 = 50% | Rate conditional on a decided label | Also report both exclusions and the total of ten. |
| Possible review outcomes | 4 / 10 to 6 / 10: 40–60% | Classification range, not a confidence interval | Resolve labels before using a binary model. |
Open each proposition to compare its diagnosis. More checks may still be needed.
No. The denominator for 50% is eight. This hides the population change and both open labels.
No. These bounds come only from possible classification outcomes; no sampling probability was calculated.
Yes. Add the conditional rate if useful and describe the panel. Annotation uncertainty differs from sampling uncertainty.
Batch: DEMO, ten selected answers Rule: citation supports the claim Decided: 4 supported and 4 unsupported Undecided: 2, passages to review Confirmed: 4/10; conditional: 4/8 Classification outcomes: 40–60%, no confidence level Next: documented adjudication of both cases
This range quantifies neither future model behaviour nor panel bias. Do not automatically replace ten by eight in every calculation.
Four successes in ten give 40% and a 95% Wilson interval of about 16.8–68.7% under the stated assumptions. The rate alone hides this width and does not establish a universal ranking.
The observed rate is zero, but the upper bound is about 27.8%. No citations in this sample does not establish that citation is impossible.
Eighteen successes in twenty give 90%. Under the stated binomial assumptions, the 95% Wilson interval is about 69.9–97.2%. If a decision requires more than 80%, the lower bound does not confirm it. Report counts and the decision rule.
Fictional case: one question is rerun twenty times in one conversation, while another panel contains twenty different needs. Equal raw rates do not make diversity or dependence comparable. Keep question and session identifiers and describe groups before calculating an overall metric.
From observation to decision
Two fictional collections give 40%: 4 successes in 10 and 40 in 100. Calculation assumes independent binary observations with stable probability. Intervals describe that assumption without establishing question representativeness.
Fictional learning case: counts and dates illustrate the method and do not describe a real assignment.
Left to right: 0 to 100%. Band: 95% Wilson interval; vertical marker: observed rate of 40%.
| Observation | Useful check | Next action |
|---|---|---|
| 4 successes in 10 | Rate: 40%; 95% Wilson interval: 16.8–68.7%. | A width of about 51.9 percentage points leaves substantial uncertainty. |
| 40 successes in 100 | Rate: 40%; 95% Wilson interval: 30.9–49.8%. | Width is about 18.9 percentage points; the protocol still needs review. |
| Same proportion, different bases | Count changes precision under the model. | Report successes, total, interval and assumptions together. |
No. Row count establishes neither independence nor diversity. Keep question, session and conditions; examine dependent response groups. If the protocol cannot support probabilistic inference, describe the panel results as descriptive.
Method: two-sided 95% Wilson interval, z ≈ 1.96. Counts are fictional; bounds are rounded to a tenth of a percentage point. NIST — Confidence intervals for proportions
Worked exercise
Fictional example: the values and documents teach the method and do not describe a real organisation.
Panel categories are disjoint. An unreadable answer is not an observed absence of citation.
If the table exceeds the screen, scroll horizontally; arrow keys work when the region is focused.
| Observation | Count |
|---|---|
| Observed citation | 12 |
| Readable without citation | 4 |
| Unable to examine | 4 |
Among 16 readable answers the rate is 12/16 = 75%. Across 20, citations range from 12 to 16, or 60–80%, depending on the unknowns. These are missing-data bounds, not a Wilson confidence interval. Processing does not turn unknown answers into observations.
After recovering four answers, replace unknowns with actual states and publish the revised summary version.
Keep counts by state, denominators, unreadability reason and exclusion rule.
No. The rate is k/n. The 95% level qualifies the interval method under the stated assumptions.
No. Overlap is not a difference test, and paired observations require appropriate treatment.
Set the rule before collection. A rate conditional on readable answers needs the access-failure count and total attempts. Do not automatically classify every inaccessible answer as an absent citation or silently exclude it.
Previne an uncertain category and describe the reason. A second reading may clarify the judgment; it should not delete the limits or impose a certainty that the case does not allow.
MEASUREMENT / VISIBILITY SOURCES
Appearing in an AI answer and receiving a website visit happen at different stages. Choose the signal that answers your question before comparing periods.
Crawling may make a page available for search.
A link can appear in an answer without receiving a click.
The site records an arrival within the limits of its measurement setup.
| Observation source | What is measured | Interpretation limit |
|---|---|---|
| Google · Generative AI in Search Console Official documentation | Link impressions in AI Overviews and AI Mode, with dimensions including page, country and device. | Days use Pacific Time (PT). Property totals can differ from the sum of page rows because of aggregation. If the report does not appear, Google says the site usually has too few impressions in these features. |
| Bing Webmaster Tools · AI Performance Official documentation · Bing announcement | Visible citations and cited pages in supported AI experiences; grounding queries are grouped. | A citation is neither a click, a ranking nor a measure of page importance. Grouped queries do not reproduce individual prompts. The queries shown reflect a sample of citation activity. |
| Website measurement | Arrivals and actions recorded under your measurement configuration. | An attributed visit does not represent every impression. Missing referrers or blocked measurement limit attribution. |
| Documented question panel | Citation presence and fidelity across a set of observed answers. | The result depends on questions, mode, language and date. It does not describe all users. |
Google AI report impressions are included in the Search Console Web scope: do not add them to the Web total. Compare visibility, citations and visits separately.
Compare the scope of two observationsDocumentation accessed on .
DOCUMENTED REFERENCE
Spread describes observations; estimator uncertainty describes sampling variability within a model. Sharing units does not make them interchangeable. R documents its SD denominator; NIST explains assumptions for mean intervals.
| Reference | Value or form | What it identifies |
|---|---|---|
| Sample variance | Σ(xᵢ − mean)² / (n − 1) | Sample variance here divides by n − 1 for retained observations. Division by n differs; record the denominator. |
| Sample SD | √variance | Square root of variance, back in original units. Variance has squared units. Large SD alone does not prove collection error. |
| IQR | Q3 − Q1 | Q3 minus Q1 describes central spread and depends on quartile convention; preserve algorithm. |
| Range | max − min | Maximum minus minimum responds strongly to extremes; not a confidence interval or future coverage probability. |
| SE(mean) | s / √n · assumptions required | Simple s/√n requires an appropriate framework, including independence. Not sufficient for every weighted survey, time series or repeated observation. |
| n < 2 | R sd(): NA | Documented R sd() returns NA for zero or one observation. Missing is not automatically zero SD. |
Mean intervals require model, level and method, not just plus/minus SD. AI citations or repeated outputs may be dependent; more rows do not ensure independent-observation uncertainty reduction.
Only after identifying their meaning. SD describes observation spread, SE mean estimation within a model. Shorter bars alone do not prove a less dispersed population.
Documentation consulted on .
DOCUMENTED REFERENCE
Five positive cases out of twenty give 25%, as do 250 out of 1,000. Under a suitable binomial model, the estimates do not have equal precision. This explorer separates the observed rate, confidence level and calculated limits.
| Reference | Value or form | What it identifies |
|---|---|---|
| Observed proportion | x / n | x counts positive cases under a fixed criterion; n counts all assessable cases. Missing or undecidable responses are not negatives: document exclusions separately. |
| Wilson score | two-sided | Wilson is a two-sided score interval. Its center generally differs from x/n and its limits stay between 0 and 1. It is neither the Wald interval nor the exact Clopper–Pearson binomial interval. |
| Confidence level | 90% · 95% · 99% | Levels are 90%, 95% and 99%, using the corresponding normal quantile. Confidence describes repeated-sampling coverage of the procedure, not a posterior probability for this one interval. |
| x = 0 | upper limit > 0 | When x = 0 and n > 0, the upper limit remains positive. No observed events do not prove impossibility. No interval is calculated when n = 0. |
| Sampling model | independent · common probability | The model assumes independent trials with a common probability. Repeated answers to one query, clustered cases and convenience samples do not automatically meet those conditions. |
Teaching exercise: no observations of a real service. Positive and negative refer only to the declared criterion.
Model assumptions: independent trials with a common probability. Selected samples and dependent repetitions do not become representative through this interval.
Enter integers with 1 ≤ n ≤ 1,000,000 and 0 ≤ x ≤ n.
Calculations stay in your browser; no inputs are transmitted and no file is created.
Results calculated · x = 5 · n = 20 · 95%
z = {"90":1.6448536269514715,"95":1.9599639845400536,"99":2.5758293035489}
p = x/n; D = 1 + z²/n
center = (p + z²/(2n)) / D
half = z × sqrt(p(1−p)/n + z²/(4n²)) / D
Wilson = [center − half, center + half]Very small p-values use scientific notation instead of displaying zero. Other results are rounded to six decimals.
| x / n | Observed rate (%) | Lower limit (%) | Upper limit (%) | Width (percentage points) |
|---|---|---|---|---|
| 0 / 5 | 0 | 0 | 43.448246 | 43.448246 |
| 1 / 5 | 20 | 3.622411 | 62.446537 | 58.824126 |
| 5 / 5 | 100 | 56.551754 | 100 | 43.448246 |
| 5 / 20 | 25 | 11.18617 | 46.870088 | 35.683918 |
| 25 / 100 | 25 | 17.545211 | 34.304464 | 16.759252 |
| 250 / 1000 | 25 | 22.41531 | 27.776028 | 5.360718 |
| 0 / 100 | 0 | 0 | 3.69935 | 3.69935 |
| 100 / 100 | 100 | 96.30065 | 100 | 3.69935 |
All examples are constructed to explain calculations, not commercial performance data.
Confidence is not the probability that a claim is true. A p-value does not measure that probability either.
Original examples, not measurements of a real service. Wilson coverage is approximate, not guaranteed to be 95% at every sample size. It corrects neither selection bias, dependence nor annotation errors and does not predict future engine changes.
Report x and n, the positive criterion, exclusions, queries and selection, versions, dates, independent unit and interval method. Citation presence does not establish citation correctness; evaluate that separately.
Documentation consulted on .
DEEPER READING / PRACTICE
Two series of answers show different quotation rates. The issue is to determine whether the gap describes a reproducible variation or simply the composition of the questions tested.
Updated on .
Examine staff, repetitions and the dependencies between observations. Separate the results by category when the questions relate to different intentions or levels of difficulty.
A precise percentage can give a false impression of solidity. A small sample or oriented selection remains limited, even accompanied by decimals and graphics.
An uncertainty note shows the number of cases, the classification rule and the variations observed. It specifies the defendable comparisons and the additional observations required.
EXAMPLE / READING THE NUMBERS
Fictional data illustrating the method; not client results or measurements of this website.
| Observation | Count | Interpretation |
|---|---|---|
| Panel questions | 40 | Fixed comparison base. |
| Answers mentioning the brand | 12 | 12 / 40 = 30% of the panel. |
| Answers describing the offer correctly | 8 | 8 / 40 = 20% of the panel; 8 / 12 ≈ 66.7% of mentions. |
These observations measure neither market share nor sales. A mention may be inaccurate. Report the panel, date, language and repetitions alongside the result.