AI Retrieval Infrastructure Visibility: How to Measure What Agents Search Before They Answer

The practical decision is which claim your records support: a page appeared in a provider's results, an agent received its content, or a user saw it cited. These are different observations and need different evidence.

Which layer are you measuring?

The original five-layer model remains useful: question, query, retrieval, answer and lifecycle. Here, retrieval means finding candidate sources; lifecycle means tracking what happens to those sources across repeat observations.

This explanatory model separates the user's question from generated searches, provider results, final answer and later citation history. It is not a recorded agent trace.

Record the exact question and relevant conversation context first. Then retain any search queries the application exposes, the provider response, the content passed to the answering model and the final answer. Across repeat runs, retain dates and consistent settings so changes remain interpretable.

Google says AI Overviews and AI Mode may use query fan-out, meaning multiple related searches across subtopics and sources. OpenAI describes ChatGPT Search rewriting a prompt into targeted queries for search providers. These disclosures explain why one question may lead to several searches. They do not reveal the complete private route for every individual answer.

What does a retrieval result actually prove?

The rule is simple: a result proves presence in the request you observed, not in every agent that could use that provider. For example, Exa’s Search quickstart documents ranked results and content outputs such as highlights or full text. Read the actual response before claiming that a page appeared in it.

This guide shows which conclusion each record supports, and where a separate answer requires its own check.

What must a trustworthy report retain?

The useful buying criterion is whether a report preserves the relationship between the request, returned evidence and answer, including where that relationship is unknown. A blended score cannot recover a missing trace.

Xtrusio saves your buyer questions, the answers and their cited pages, and links each fix to the page it changed. Standard monitoring uses API-returned answers; it is not an exact reproduction of every consumer interface.

Xtrusio does not offer direct Exa integration or tracing of private consumer backends; the metrics below are definitions you can apply in your own controlled tests. For the question-level workflow, see Xtrusio question-level workflow.

Profound's September 4, 2026 changelog announced Exa support in Answer Engine Insights, with activation through the Profound team for enterprise customers. The same dated section announced broader Citation Decay availability.

How do you link retrieval to the answer?

Use two separate designs. In a controlled application, your team can preserve the retrieval call and the evidence actually passed to the answering step. In a consumer comparison, your team observes the public answer and compares its citations with a separately collected provider response. The second design measures overlap, not transfer.

For a controlled run, assign one run identifier to the user question, search request and response. Retain the raw returned URLs and passages, any selection or truncation, the exact source context supplied to the model, the final answer and its citations. If the agent searches again or opens additional pages, those events need their own records under the same run.

This distinction follows the architecture illustrated by OpenAI's retrieval documentation: an application retrieves results and supplies those results with the original question to a model. That API pattern does not describe the private backend of consumer ChatGPT, and does not establish that Exa is its provider.

Consider a hypothetical agent answering, “Does this contract renewal policy require 60 days' notice?” A search response contains a policy page, but the application passes only a short excerpt to the model. If the notice clause was omitted, the page's presence in the search response does not show that the model received the clause. Preserve what was passed, not just the result title.

If a separate ChatGPT answer cites the same policy page, report exact-page overlap between the two observations. Similar prompts and close timestamps do not join their execution paths. A full controlled trace can establish what entered your application and what it output; it still does not reveal which text had causal influence inside the model.

The renewal-policy example is hypothetical. Use the same-run source and answer record when evaluating your own application; compare consumer citations separately.

Which proposed metrics are useful?

The following three definitions are proposed Xtrusio operating metrics. They are not industry standards, benchmarks or implemented product features. Choose the scope and outcome before calculating them.

Retrieval Surface Coverage

Definition: completed provider observations containing a tracked exact page within the selected result depth, divided by all completed observations for that provider and configuration, multiplied by 100. Name the scope, such as Exa top-10 coverage. Do not mix different providers or result depths into an unexplained percentage.

Answer Transfer Rate

Definition: eligible controlled runs with a defined answer outcome, divided by eligible controlled runs that received tracked source content, multiplied by 100. Eligibility requires the source-to-answer input trace and captured final answer from the same run. Choose one outcome, such as citation of the exact supplied page; report mere brand mention separately.

This measures an observed input-to-output relationship in your controlled application, not causal importance or conversion through a consumer product's hidden backend. If only independent provider results and consumer citations exist, leave transfer unavailable and report URL overlap instead.

Citation Persistence

Definition: completed eligible repeat windows retaining the exact URL citation, divided by completed eligible windows after the first observed citation, multiplied by 100. Define the window and cohort consistently. A missing answer or uncaptured window is unknown, not a citation absence.

The preserved graphic assumes all 120 planned provider observations completed as separate controlled runs: 40 questions across three surfaces, with 34 containing tracked evidence. That yields a pooled teaching total of 34/120 = 28.3% across the three stated surfaces. Report each provider and result depth separately for comparisons; this pooled illustration is not provider-specific coverage. Its 18/34 = 52.9% transfer example additionally assumes those 34 runs passed that tracked evidence to the answering step and have a complete same-run input trace and final answer, with 18 meeting the defined citation outcome. Six of eight completed repeat windows retain the citation, giving 75%. These are invented teaching numbers, not a scan or customer result.

What changes when captures are incomplete?

Report completion alongside each result. The following separate hypothetical cohort shows how failures affect the denominators; it does not revise or claim the all-complete graphic's numbers.

The 41.7% applies only to the 24 fully observed joined runs. Missing traces could be systematically different, so do not report it as transfer across all 30 or a forecast for future requests. A complete provider response with no tracked page is a negative coverage observation; a request failure supplies no such result.

Which metadata makes a repeat test interpretable?

Keep the question, observable rewritten query, request timestamp, provider, search type, result depth, filters and returned URLs. Preserve raw addresses before any normalization, alongside the actual passages passed to the model. Record publication dates and crawl or fetch metadata only when supplied and meaningful; leave unavailable values unknown.

Exa's current best practices distinguish publication recency from content freshness. The publishedDate field is not your observation time or proof of a crawl timestamp. The contents.maxAgeHours setting governs extracted content freshness; zero requests a fresh fetch, while negative one restricts content to the cache. It is not a publication-date filter.

The same guide says startCrawlDate and endCrawlDate request fields are ignored. Use the documented publication filters for publication dates; do not rename them crawl filters. Retain the actual request settings, SDK version and available request identifier so a later schema or default change is visible.

For consumer answers, record the actual question, interface, mode, date, available account/locale context, complete answer and opened citation destinations. Hidden rewrites and unavailable provider records remain unknown. Matching domains can support a broad source comparison; they cannot replace exact-page matching or claim support in the cited passage.

Why retrieval loss and citation loss need different investigations

The curves are an invented lifecycle shape. They illustrate how provider presence and citation presence can diverge, not an observed trend or attribution to a content change.

If provider presence and citations both fall, investigate access, request changes, freshness and competing sources. If provider presence remains but citations fall, inspect the context actually passed, whether it directly answers the question and whether the final claim is supported. When only separate consumer comparisons exist, treat these as hypotheses rather than a diagnosis of its private source-selection stage.

Profound's Citation Decay definition, published September 29, 2026, uses a 14-day rolling citation-share average. Decay requires share at or below half its peak for 14 straight days through the current observation, without a bounce above the threshold. Eligibility also requires at least 28 live/citable days, five citations in the window, a positive peak share and five raw citations in the peak 14-day window. This is Profound's product rule, not the proposed persistence formula or a universal refresh deadline.

For a first report, decide which relationship you can observe before collecting more percentages. Ask Xtrusio for a scoped free evaluation of the buyer questions, available answer evidence and required retrieval instrumentation.