Benchmarks
Measured, with the method
What we measured, how, and on what data. Two kinds of runs feed this page: end-to-end POST /search calls against the live web through a local ContextFetch stack, and offline engine runs on a synthetic corpus written for this repo. Each block names its source file, the method, the sample size and the unit. Competitor latency and quality appear only when they were measured.
live web real queries, real pages · synthetic docs and fixtures written for this repo
1,644ms p50
Search, cold
5 live-web queries, query-cache miss (cache = miss); URL-cache state not recorded, client-observed. Max 2,300 ms.
9ms p50
Same query, cached
The same 5 requests repeated, answered from the query cache. Max 1,842 ms.
69% mean
Context tokens saved
prunedContext vs the whole doc, 29 pairs. Synthetic docs, not live pages.
1.98x
Stored compression
zstd with dictionary, plus base64 in the vector payload. Synthetic held-out docs.
Production API, called from another continent
The same five queries against the live API at api.contextfetch.dev, sent from a client in Florida to the server in Helsinki, so every time includes a transatlantic round trip. Each query is sent once (cold) and once more (repeat, answered from the query cache). 10 results per query with page text and highlights, timed by the client.
Source: results_phase2_prod.json · Run 2026-10-17
| Group | Cold | Repeat |
|---|---|---|
| python asyncio timeout | 2,204 ms | 150 ms |
| stripe webhook signature verification | 3,311 ms | 153 ms |
| sec edgar apple 10-k revenue table | 2,809 ms | 181 ms |
| qdrant binary quantization benchmarks | 3,141 ms | 143 ms |
| react 19 useActionState example | 3,531 ms | 282 ms |
| Query | Cold ms | Repeat ms | Results | 500+ chars |
|---|---|---|---|---|
| python asyncio timeout | 2,204 | 150 | 10 | 3 |
| stripe webhook signature verification | 3,311 | 153 | 10 | 9 |
| sec edgar apple 10-k revenue table | 2,809 | 181 | 10 | 2 |
| qdrant binary quantization benchmarks | 3,141 | 143 | 10 | 7 |
| react 19 useActionState example | 3,531 | 282 | 10 | 10 |
End-to-end search latency on one machine
Each query is sent to POST /search with page text and highlights, once as its first request (cold: query-cache miss (cache = miss); URL-cache state not recorded) and once repeated (cached), through the full local stack. The benchmark flushes no cache, so pages fetched by an earlier run may still be in the URL cache. Times are measured by the client. Results counts and the number of results with at least 500 characters of page Markdown come from the same responses.
Source: results_phase2_after.json · Command: .venv/bin/python benchmarks/five_query_bench.py <BASE> <CA_FILE> <before|after> · Date, machine and load are not recorded in this file.
| Query | Cold ms | Cached ms | Results | With 500+ chars Markdown |
|---|---|---|---|---|
| python asyncio timeout | 2,300 | 7 | 10 | 10 |
| stripe webhook signature verification | 1,704 | 14 | 10 | 7 |
| sec edgar apple 10-k revenue table | 1,633 | 1,842 | 0 | 0 |
| qdrant binary quantization benchmarks | 1,460 | 9 | 10 | 9 |
| react 19 useActionState example | 1,644 | 6 | 10 | 5 |
Where a cold search spends its time
Server-side stage timings from the response's contextfetch.timingsMs field, in ms. Stages can overlap, so they need not add up to the total. Bars share one scale.
python asyncio timeout
recall490fetch635rerank1,400total2,284stripe webhook signature verification
recall372fetch953rerank596total1,696sec edgar apple 10-k revenue table
recall902fetch0rerank724total1,630qdrant binary quantization benchmarks
recall322fetch531rerank729total1,454react 19 useActionState example
recall353fetch753rerank707total1,636
Before and after tuning
The same queries and request body against the engine before and after the Phase 2 changes (parallel recall sources, a fetch time budget and caches). Cold runs only. Cold p50 went from 4,140 ms to 1,644 ms.
Source: results_phase2_before.json · Command: .venv/bin/python benchmarks/five_query_bench.py <BASE> <CA_FILE> <before|after> · Date, machine and load are not recorded in this file.
| Query | Cold ms before | Cold ms after | Results before | Results after | 500+ chars before | 500+ chars after |
|---|---|---|---|---|---|---|
| python asyncio timeout | 6,356 | 2,300 | 10 | 10 | 7 | 10 |
| stripe webhook signature verification | 4,949 | 1,704 | 5 | 10 | 4 | 7 |
| sec edgar apple 10-k revenue table | 3,112 | 1,633 | 5 | 0 | 2 | 0 |
| qdrant binary quantization benchmarks | 4,140 | 1,460 | 5 | 10 | 4 | 9 |
| react 19 useActionState example | 3,883 | 1,644 | 5 | 10 | 4 | 5 |
Recall sources per query
Candidates each recall source returned for the cold run of each query, from contextfetch.sources. A blank cell means the source was not queried for that query.
Source: results_phase2_after.json · Command: .venv/bin/python benchmarks/five_query_bench.py <BASE> <CA_FILE> <before|after> · Date, machine and load are not recorded in this file.
| Query | stackoverflow | wikipedia | hn | searxng | github | local | arxiv |
|---|---|---|---|---|---|---|---|
| python asyncio timeout | 10 | 0 | 7 | 85 | 10 | 0 | |
| stripe webhook signature verification | 10 | 1 | 4 | 21 | 10 | 0 | |
| sec edgar apple 10-k revenue table | 0 | 0 | 0 | 10 | |||
| qdrant binary quantization benchmarks | 0 | 0 | 0 | 2 | 1 | 10 | 10 |
| react 19 useActionState example | 2 | 1 | 0 | 3 | 5 | 10 |
Token pruning
29 fixed question/doc pairs. Each doc is chunked, cross-encoded against the question and pruned with the API defaults; savings compare the pruned context with the whole doc. The docs are synthetic, written for this repo, not live pages, and the questions were written by the same author. Docs average 760.8 tokens and 5.66 chunks.
Source: results.json · Command: .venv/bin/python benchmarks/run_benchmark.py · Run 2026-10-16T22:09:29Z
| Tokens saved | % |
|---|---|
| Mean | 69.5 |
| Median | 79 |
| Min | 13.1 |
| Max | 85.5 |
Answer kept, % of pairs
top-1 / top-3: the answer-bearing chunk is the first or among the first three highlights. pruned: the pruned context still contains it. random: the chance a random chunk is the right one.
Engine component latency
Wall-clock time around single calls after a warm-up, on the synthetic corpus and hand-written HTML fixtures.
Source: results.json · Command: .venv/bin/python benchmarks/run_benchmark.py · Run 2026-10-16T22:09:29Z
| Operation | n | p50 ms | p95 ms | Unit |
|---|---|---|---|---|
| embed_per_doc | 150 | 109.6 | 167.49 | one page (title + first 2,000 Markdown chars) per call |
| rerank_per_20_chunks | 87 | 501.45 | 573.3 | one query x 20 random corpus chunks per call |
| prune_per_page | 29 | 136.4 | 173.86 | chunk + cross-encode + select one corpus doc |
| html_to_markdown | 100 | 4.6 | 6.97 | engine page_to_result (trafilatura) per fixture page |
| Fixture (hand-written) | HTML bytes | Markdown chars | n | p50 ms | p95 ms |
|---|---|---|---|---|---|
| api-ref-ledgerline-payments.html | 9,818 | 3,746 | 25 | 6.22 | 7.18 |
| blog-postgres-partial-indexes.html | 11,725 | 3,303 | 25 | 3.19 | 3.27 |
| docs-nginx-rate-limiting.html | 11,597 | 2,773 | 25 | 3.28 | 3.32 |
| qa-asyncio-blocking-timeout.html | 8,804 | 1,796 | 25 | 6.76 | 7.13 |
Model cold start: embedding model load + first call 104 ms; cross-encoder load + first call 87.1 ms.
Storage and vector recall
zstd level 19 on 10 held-out synthetic docs, dictionary trained on 40 others. The stored ratio also counts the base64 encoding used in the vector payload; use it for capacity estimates. Vector bytes: raw bytes = vectors x bytes_per_vector; float32 = dim*4, uint8 = dim, 1-bit = ceil(dim/8); a projection by arithmetic, not a measurement.
Source: results.json · Command: .venv/bin/python benchmarks/run_benchmark.py · Run 2026-10-16T22:09:29Z
| Codec | Ratio | Bytes |
|---|---|---|
| Raw Markdown | 1x | 29,187 |
| zstd + dictionary | 2.64x | 11,055 |
| zstd, no dictionary | 2.06x | 14,152 |
| Stored (dictionary + base64) | 1.98x | 14,752 |
| Format | Bytes per vector (dim 384) | Raw GB at 250,000,000 vectors |
|---|---|---|
| float32 | 1,536 | 384 |
| uint8 | 384 | 96 |
| 1-bit | 48 | 12 |
| Query set | Queries | Corpus vectors | binary only | binary top-20 + float rescore | binary top-40 + float rescore |
|---|---|---|---|---|---|
| held-out doc chunks vs train-doc chunks | 54 | 206 | 0.522 | 0.717 | 0.861 |
| natural-language questions vs all chunks | 29 | 260 | 0.628 | 0.79 | 0.903 |
Machine
The machine the offline engine run used. The end-to-end search files do not record their machine.
Source: results.json · Command: .venv/bin/python benchmarks/run_benchmark.py · Run 2026-10-16T22:09:29Z
| Item | Value |
|---|---|
| CPU | Apple M2 |
| Logical CPUs | 8 |
| RAM | 24 GiB |
| Load average at start (1 / 5 / 15 min) | 7.7 / 10.4 / 14.6 |
| OS | macOS-27.0.1-arm64-arm-64bit-Mach-O |
| Python | 3.13.3 |
| onnxruntime | 1.31.0 |
| fastembed | 0.9.0 |
| zstandard | 0.25.0 |
| trafilatura | 2.3.1 |
Competitors
Run the numbers on your own queries
The benchmark scripts are in the repository. Or try the live API in the browser, no key needed.