Measure useful AI-agent latency

When does the result become usable?

First displayed content, end of generation and human acceptance are different events. A useful comparison states timing boundaries and expected output. This protocol ranks no vendor; it organises reproducible local measurements.

1. Define boundaries

   Specify request sent, first content, tools completed, output ready or review finished. Use a duration-appropriate clock. Do not subtract clocks on different machines without known synchronisation.

2. Fix the scenario

   Keep input, instructions, version, permissions and cache state. Separate first and repeated calls. Record external-tool time rather than attributing everything to the model.

3. Retain failed trials

   Record durations, attempts, retries and timeouts. Publish count and distribution, not just the best result. An abandoned task remains visible even without a completion duration.

4. Compare accepted quality

   Apply the same acceptance criterion. Count corrections and review time. Fast invalid output is not useful time saved; state human effort.

Situations and decisions

Typical situations for preparing a check. They do not describe completed assignments or actual observations.

Record to retain

• Timing boundaries and units
• Versions, input and cache state
• Durations, failures and retries
• Quality criterion and human time

Official references

• W3C — User Timing Level 2
• NIST — Generative AI Profile

Related protocols

• Evaluate an AI agent before release
• Document a sample before publishing a measurement
• Map AI system dependencies

Checks before reaching a conclusion

• Are boundaries identical?
• Is cache state recorded?
• Are failures visible?
• Is quality judged consistently?

Stage timings linked to quality, failures and acceptance effort.