Benchmark AI Search APIs on the Same 25 Research Questions

*Research-based editorial guidance, checked October 21, 2026. No independent product benchmark or hands-on performance rating is claimed.*

A benchmark you can actually repeat

This guide provides a question pack and recording method for testing search APIs against your workload. It contains no provider rankings or measured performance results. Product accounts and a controlled run are needed before making those claims.

Download the 25-question CSV. The pack contains five questions each for current pricing, technical documentation, date-sensitive updates, conflicting evidence and missing or inaccessible evidence. Replace date-sensitive targets when you run it and retain the original question version.

Freeze the protocol first

Record run date, question wording, provider endpoint, request parameters, result count, domain filters, timeout and retry policy. Save the provider configuration and the downstream answer model. Use the same answer prompt and model for all providers in the end-to-end track.

Run two tracks: discovery-only measures returned URLs; end-to-end measures whether a pipeline produces a supported answer. A search-only endpoint should not be judged against a research endpoint without disclosing the extra work, latency and cost. The Exa/Tavily/Firecrawl comparison explains the role differences.

Establish expected evidence

Before running providers, have a reviewer identify authoritative sources and the facts required for a correct answer. Retain snapshots or dated notes where permitted. For fresh facts, a page that was correct last month may fail today. For conflicting sources, the answer should identify the conflict rather than silently average incompatible numbers.

Leave some questions genuinely unanswered. The correct output for an unpublished price or inaccessible document can be “not established.” This tests whether a pipeline produces plausible but unsupported detail to avoid an empty response.

Score four dimensions separately

Use 0 for unsupported, 1 for incomplete and 2 for fully supported answers. Have reviewers inspect passages, not just whether a citation link exists. Score discovery relevance separately so a synthesis failure is not wrongly attributed to search.

Measure latency and cost honestly

Record end-to-end wall time including retries. With only 25 questions, a tail percentile is unstable; report individual slow cases and the median, then expand before making production latency commitments. Repeat in a second time window and disclose cache behavior.

Sum actual provider and model charges across successful and failed calls. Divide by accepted answers. If none are accepted, report no accepted-answer cost rather than dividing by zero. Keep reviewer minutes visible and apply your own labor assumption separately.

Turn a pilot into publishable evidence

Retain question IDs, raw response files, endpoint settings, source dates, scoring notes and disagreements. Publish anonymized outputs only where permitted. Report the workload boundary and sample size beside any conclusion; do not call a small custom pack a universal benchmark.

Choose a provider only after the relevant track clears your preset support and cost thresholds. If results are close, operational simplicity and failure recovery may decide the purchase. Re-run after material endpoint, prompt or source changes.