The Cheapest Token You Never Send: Agentic AI Search on MongoDB + Voyage AI

TL;DR

The most expensive part of a production agentic AI application usually isn't the model you call — it's what you feed the model. Every chunk of retrieved context is an input token you pay for on every request, and low-relevance context doesn't just cost money; it dilutes the answer. This post walks through an agentic AI search pipeline built on MongoDB Vector Search and Voyage AI embeddings and re-rankers, and shows how each stage — embed, retrieve, rerank, iterate — simultaneously lowers token cost, speeds up search, and improves answer quality. The through-line is simple: better retrieval is the cheapest lever you have.

1. The hidden token tax in AI search

A typical first-generation AI search app works like this: embed the user's question, pulls back the top N nearest chunks, staples them into the prompt, and sends it all to the LLM. It works in a demo. Then it hits production, and three problems show up at once:

• Cost: If you retrieve 20 chunks of ~500 tokens each "to be safe," you're paying for ~10,000 input tokens on every call — most of which the model doesn't need.

• Latency: Bigger prompts take longer to transmit and process, and a bolt-on vector database adds network hops and a second system to keep in sync.

• Quality: Counterintuitively, more context often produces worse answers. Irrelevant passages bury the signal, and models are prone to "lost in the middle" effects, where key facts in a long context get ignored.

These aren't three separate problems. They're one problem — retrieval precision — viewed from three angles. Fix retrieval, and all three improve together. That's the opportunity, and it's what turns basic keyword lookup into true agentic AI search: a system that reasons about what to retrieve, retrieves precisely, and acts on it.

2. The pipeline at a glance

Here's the shape of a cost-efficient agentic AI search pipeline where the operational data, the vector index, the metadata, and even the embedding and reranking calls all live in one platform:

The key architectural idea: consolidation. Because MongoDB stores your application data, the vector embeddings, and the metadata together — and Voyage AI's industry-leading embedding and reranking models are available natively on Atlas via the Embedding and Reranking API — you avoid the data-sync pipelines, extra network round-trips, and glue code that come with bolting a separate vector database onto your primary store.

Let's walk the pipeline stage by stage and look at where the tokens (and milliseconds, and dollars) actually go.

3. Stage 1 — Embeddings that cost less to store and query

Retrieval quality starts with the embedding model, but cost at scale starts with how those vectors are stored. Two Voyage AI + MongoDB capabilities matter here.

Flexible dimensionality. Voyage AI text embedding models support a range of output dimensions (from 256 up to 2048), so you can trade a little accuracy for a lot of index footprint when your workload needs it. See the Voyage models overview for the current lineup.

Automatic quantization. MongoDB Vector Search can automatically quantize float embeddings to compact int8 (scalar) or binary representations directly in the index — you just declare it, and both your stored vectors and incoming query vectors are quantized at query time with no change to your query code. Per the Vector Quantization docs, scalar quantization reduces RAM usage by roughly 3.75x and binary by roughly 24x. Less RAM per vector means more vectors per node, lower cost, and faster search.

(Automatic quantization requires MongoDB 8.0 or later.) For a deep dive on the accuracy/performance trade-offs of each quantization type, see Scaling Vector Search With MongoDB Atlas Quantization & Voyage AI Embeddings.

4. Stage 2 — Faster, more precise candidate retrieval

Semantic (vector) search is great at meaning but can miss exact keywords, product codes, or names. Lexical (full-text) search nails those but misses paraphrases. Running them separately and merging results in your app is slow and awkward.

MongoDB does this in a single query with the $rankFusion aggregation stage, which implements Reciprocal Rank Fusion (RRF) natively — combining a $vectorSearch pipeline and a full-text $search pipeline server-side:

Two token-and-latency wins here. First, one round-trip instead of two systems and a client-side merge. Second — and this is easy to overlook — you can attach metadata filters (tenant, date range, document type) so you never even retrieve, embed-compare, or pay to process data that couldn't be relevant. For more on hybrid search, see Harness the Power of Atlas Search and Vector Search with $rankFusion.

At this stage, we deliberately retrieve broadly — a generous candidate set optimized for recall. We'll tighten it in the next stage, cheaply.

5. Stage 3 — Reranking to shrink the context window

This is where the token savings become dramatic.

A reranker takes the broad candidate set from Stage 2 and reorders it by true relevance to the query using a more powerful cross-encoder model, then you keep only the top few. Voyage AI rerankers are available natively on Atlas through the same Embedding and Reranking API, so this is one call, not another vendor integration.

The pattern is retrieve broad, rerank sharp, send small:

Because the reranker runs against candidates you've already narrowed with quantized vector search and metadata filters, it's operating on a small set — so it's fast — and its output lets you cut the LLM's input context by a large factor. Fewer input tokens, lower latency, and typically a better answer because the model isn't wading through noise. For the retrieval-quality rationale behind this approach, see Rethinking Information Retrieval in MongoDB with Voyage AI.

Note: Voyage AI's current recommended reranker is rerank-2.5 (32K token context), with rerank-2.5-lite available for latency-sensitive workloads.

6. Stage 4 — The agentic loop

This is what makes the search agentic rather than a single retrieve-then-answer pass. An agent decides whether it needs to retrieve, issues one or more retrieval tool calls, evaluates what it got back, and only then generates, sometimes looping to refine the query. Search becomes an action the agent takes and reasons over, not a fixed preprocessing step.

Figure 2. The agentic AI search loop on MongoDB Atlas for efficient, iterative retrieval.

This loop is exactly where a consolidated data platform pays off. Every iteration of the loop is a round-trip; when your operational data, vectors, metadata, embeddings, and reranking all live in Atlas, each iteration is fewer network hops and less glue code than coordinating a separate vector store, a separate embedding service, and a separate reranking service. Fewer, sharper retrievals per loop also mean fewer tokens burned per agent step — and agent steps compound.

7. Putting it together: an illustrative token-cost example

The numbers below are illustrative arithmetic, not a benchmark — plug in your own chunk sizes, model, and pricing.

Say each retrieved chunk is ~500 tokens, and you make 100,000 agentic search calls a month:

That's an 80% reduction in retrieved-context input tokens in this illustration — before counting the latency win from smaller prompts and the quality win from removing noise. The reranking call itself has a cost, but it operates on a small candidate set and is typically a small fraction of what you save on the LLM's input tokens. The exact ratio depends on your models and pricing; the direction is reliable.

8. When to reach for what

9. Wrap-up

Token cost, latency, and answer quality in an agentic AI search system are not three problems — they're one problem, retrieval precision, wearing three hats. By treating the pipeline as a cost-optimization system rather than a pile of features, you can attack all three at once:

• Embed and quantize so vectors cost less to store and search.

• Hybrid retrieve in one query, with metadata filters so you never pay to process irrelevant data.

• Rerank and send small so the LLM gets a tight, high-signal context — the single biggest token-cost lever.

• Loop agentically over a consolidated platform so each step has fewer hops and less glue.

The cheapest token, and the one least likely to mislead your model, is the one you never send.