How to Estimate the Infrastructure Cost of a Production RAG Pipeline

In our illustrative model, a production RAG pipeline costs about $1,350 a month for an internal assistant answering 100,000 questions, $6,550 for a support assistant answering 500,000, and $28,350 for enterprise search answering 2 million. That is infrastructure only, before the engineers who run it.

The surprise for most teams is where the money goes. LLM generation is 78-84% of the bill in every scenario, because each answer sends several thousand tokens of retrieved text to the model. Embedding the whole corpus costs between about $1 and $115, once. The vector database is often the smallest line, until the index grows large enough that every query scans tens of gigabytes.

So the biggest savings come from sending fewer, better chunks to the model, not from shopping for a cheaper vector store. The method below shows each component and its formula, so you can plug in your own corpus size and query volume.

The six cost components of a RAG pipeline

A RAG pipeline has two halves: building and maintaining the index, and answering each query. Most of the cost sits in the second half.

Index side (mostly one-time, plus churn):

• Ingestion and chunking. Parsing documents, splitting them into chunks and attaching metadata. Usually compute you already run.
• Embedding the corpus. Converting every chunk into a vector, once at the start and again whenever content changes or you switch embedding models.
• Vector storage and writes. Storing the vectors and their metadata, and paying to write or overwrite them.

Query side (every request):

• Query embedding and vector search. Embedding the user's question and searching the index.
• Reranking. Re-scoring the top results with a more accurate model, so only the best chunks reach the LLM. Optional but common.
• LLM generation. Sending the question plus retrieved chunks to the model and generating the answer.

On top of these sit operating costs that teams often leave out: evaluation, observability, re-indexing projects and the engineers who maintain the pipeline. Those are covered later.

Ingestion and embedding costs

Embedding a corpus is far cheaper than most teams expect. At current API prices, even a five-billion-token document collection costs about $115 to embed.

Sourced prices: OpenAI prices text-embedding-3-small at $0.02 per million tokens and text-embedding-3-large at $0.13 per million. Both support a dimensions parameter that shortens the stored vector; text-embedding-3-large produces up to 3,072 dimensions (OpenAI, Jan 2024; current price confirmed on OpenAI's model page).

The formula:

\text{Embedding cost} = \frac{\text{corpus tokens} \times (1 + \text{overlap})}{10^6} \times \text{price per million tokens}

Worked examples (illustrative, text-embedding-3-small, 15% overlap):

• 50 million tokens (a mid-sized internal knowledge base): 50 × 1.15 × $0.02 = $1.15
• 200 million tokens (a product's full help center, tickets and manuals): $4.60
• 5 billion tokens (an enterprise document archive): 5,000 × 1.15 × $0.02 = $115

With text-embedding-3-large the same figures are 6.5 times higher, still small.

Where the real ingestion cost sits:

• Parsing. PDFs with tables, scanned documents and slide decks need extraction or OCR, which can cost more in compute and engineering time than the embeddings themselves.
• Re-embedding when you change models. Switching embedding models means re-embedding everything, because vectors from different models cannot be mixed in one index. The API cost is small; the migration work and testing are not.
• Ongoing churn. At 10% of content changing per month, re-embedding costs pennies to about $12 a month in our scenarios. The pipeline that detects and processes those changes is the real cost.

Vector storage and search

Managed vector databases are cheap for small indexes and get expensive in one specific way: on usage-based pricing, query cost can grow with index size, not just with query count.

Sourced pricing: Pinecone as an example. Pinecone's on-demand Standard plan has a $50 monthly minimum, then charges $0.33 per GB-month of storage, $4 to $4.50 per million write units and $16 to $18 per million read units, varying by cloud and region (Pinecone pricing).

Its cost documentation sets the rules (Pinecone docs):

• Storage per record: ID bytes + metadata bytes + (dimensions × 4 bytes) for dense vectors
• Writes: 1 write unit per KB of upsert request, minimum 5 per request
• Queries: 1 read unit per GB of namespace queried, minimum 0.25 per query, regardless of how many results are returned

Other managed and self-managed options (pgvector on an existing Postgres, open-source vector databases, search engines with vector support) price differently, often by instance size rather than usage. The sizing math below still applies.

Sizing an index (illustrative). One chunk stored at 1,536 dimensions with 1,000 bytes of metadata: 8 + 1,536 × 4 + 1,000 = 7,152 bytes.

• 100,000 chunks: about 0.72 GB
• 400,000 chunks: about 2.86 GB
• 10 million chunks: about 71.5 GB

Why index size drives query cost: Under the read-unit rule, a query against a 71.5 GB namespace costs 71.5 read units. At $16 per million, that is about $0.0011 per query, or $2,289 for 2 million queries a month. The same query against a 0.72 GB index costs one-hundredth as much.

The practical fix is partitioning. If each query only needs one department's, customer's or product line's documents, store them in separate namespaces so each search scans only the relevant slice.

Query-time costs: embedding, reranking and generation

Every question triggers three paid steps. The first is negligible, the second is moderate, and the third dominates.

Query embedding

A 30-token question at $0.02 per million tokens costs $0.0000006. Even 2 million questions a month cost about $1.20.

Reranking

Rerankers re-score the top 20 or so search results so only the best few reach the LLM. Pricing is in flux: Cohere previously charged $2.00 per 1,000 searches for Rerank 4 Fast and $2.50 for Rerank 4 Pro. According to UsagePricing's capture of Cohere's rate card, Rerank 5 (released October 2026) moved to per-token billing at a reported $0.02 per million tokens for Fast and $0.05 for Pro (UsagePricing, Oct 2026). Verify on Cohere's own pricing page before budgeting.

Under per-token pricing, cost depends on how much text you rerank: 20 chunks of 500 tokens is 10,000 tokens per query. Short snippets get cheaper; long passages may not. This article uses the older $2 per 1,000 searches as a conservative estimate.

LLM generation

This is the line that matters. Each answer sends the system prompt, the question and the retrieved chunks to the model.

\text{Generation cost} = \frac{\text{queries} \times \text{input tokens}}{10^6} \times p_{in} + \frac{\text{queries} \times \text{output tokens}}{10^6} \times p_{out}

With 5 chunks of 500 tokens, plus a 500-token system prompt and the question, a typical request carries about 3,500 input tokens and returns about 400 output tokens. At $2 and $10 per million (Claude Sonnet 5.5 standard, as used in our batch vs real-time article):

• Per query: 3,500 × $2 ÷ 10⁶ + 400 × $10 ÷ 10⁶ = $0.007 + $0.004 = $0.011
• Per 100,000 queries: $1,100

Retrieved context, not the answer, is most of the input. Every extra chunk adds about $0.001 per query at these prices.

Cost model: three RAG deployments (illustrative)

In all three scenarios, LLM generation is 78-84% of monthly infrastructure cost and the vector database is under 10%. All use the assumptions above.

Scenario A: internal knowledge assistant

50 million-token corpus (100,000 chunks, 0.72 GB index); 100,000 questions a month.

• One-time setup: embedding $1.15 + initial writes $2.86 = about $4
• Vector database: storage $0.24 + reads $1.14 + churn writes $0.29 = $1.67, billed at the $50 minimum
• Content churn re-embedding: $0.11
• Query embedding: $0.06
• Reranking: 100 × $2 = $200
• LLM generation: 100,000 × $0.011 = $1,100
• Monthly total: about $1,350, of which generation is 81%

Scenario B: customer support assistant

200 million-token corpus (400,000 chunks, 2.86 GB index); 500,000 questions a month.

• One-time setup: embedding $4.60 + initial writes $11.44 = about $16
• Vector database: storage $0.94 + reads $22.89 + churn writes $1.14 = $24.97, billed at the $50 minimum
• Content churn re-embedding: $0.46
• Query embedding: $0.30
• Reranking: 500 × $2 = $1,000
• LLM generation: 500,000 × $0.011 = $5,500
• Monthly total: about $6,551, of which generation is 84%

Scenario C: enterprise document search

5 billion-token corpus (10 million chunks, 71.5 GB index in one namespace); 2 million questions a month.

• One-time setup: embedding $115 + initial writes $286 = about $401
• Vector database: storage $23.60 + reads $2,288.64 + churn writes $28.61 = $2,341
• Content churn re-embedding: $11.50
• Query embedding: $1.20
• Reranking: 2,000 × $2 = $4,000
• LLM generation: 2,000,000 × $0.011 = $22,000
• Monthly total: about $28,354, of which generation is 78%

What changes the picture

Partitioning Scenario C into ten equal namespaces would cut vector read cost about tenfold, to roughly $229 a month. Per-token reranking, if the reported Rerank 5 prices hold, would cut reranking to about $20, $100 and $400 a month in A, B and C. A cheaper model for generation moves the total far more than any retrieval change. Test quality first.

Costs teams forget

The infrastructure bill above is the smaller part of running RAG in production. People and process add more.

• Engineering. Someone owns ingestion pipelines, chunking strategy, retrieval tuning, access controls and upgrades. Using our maintenance article's figures, half a mid-level AI engineer costs about $6,336 a month from a Latin America team ($76,032 a year) or $11,913 in-house in the US ($142,955 a year). In Scenario A that is several times the infrastructure bill; in Scenario B it roughly equals it (Latin America) or nearly doubles it (in-house).
• Evaluation. RAG quality fails quietly: the wrong chunk is retrieved and the model answers confidently anyway. Budget a test set of real questions with known correct sources, rerun on every change, plus human review of sampled answers.
• Re-indexing projects. Changing chunk size, embedding model or metadata schema means rebuilding the index. The API bill is small; the engineering and testing time is not.
• Access control. Enterprise RAG must respect who is allowed to see which document. Mapping permissions into metadata filters and keeping them in sync is real work, and a mistake is a data leak.
• Observability. Tracing which chunks were retrieved for each answer is how you debug wrong answers. See our maintenance article for tool pricing.
• Document parsing at scale. Tables, scans and slides may need specialist extraction services or OCR.
• Environments. A separate staging index for testing changes doubles storage and write costs, though at these prices that is usually small.

Levers that cut RAG cost without hurting quality

Because generation dominates, the most effective savings reduce the tokens sent to the model. Test every change against your evaluation set.

• Retrieve fewer, better chunks. Reranking 20 results and sending the best 3 instead of 5 cuts about 1,000 input tokens per query: roughly $2 per 1,000 queries at the prices above, often with no loss in answer quality.
• Right-size chunks. Smaller chunks mean less irrelevant text per retrieved passage. Too small and context is lost; test a few sizes.
• Cache the system prompt. It repeats on every request. Prompt caching cuts its cost sharply on providers that support it (see our batch vs real-time article).
• Route by difficulty. Send simple, high-confidence questions to a smaller, cheaper model and reserve the larger one for hard cases.
• Cache frequent answers. Many support and internal questions repeat. Serving a stored answer for a repeated question costs nothing in generation.
• Cap output length. Ask for concise answers with citations, not essays. Output tokens cost five times input tokens at the prices used here.
• Partition the index. Separate namespaces by tenant, department or product line, so each search scans only what it needs.
• Shorten embeddings if quality allows. Fewer dimensions mean smaller storage and, on read-unit pricing, cheaper queries. Test recall first.
• Batch offline work. Re-indexing, evaluation runs and summary generation can use batch APIs at half price.

Where CodingCops fits

Disclosure: the author works with CodingCops. This section uses only the company's published descriptions and includes no CodingCops pricing, client results or savings claims. CodingCops is a Chicago-headquartered software company (Built In Chicago). Its company profile lists AI and machine learning, cloud and DevOps, custom software development, and dedicated engineering teams among its services (Outsource Accelerator), and it offers engagements for specific projects, short-term initiatives or long-term development (codingcops.com). The ongoing engineering behind a RAG pipeline, ingestion, retrieval tuning, access control and evaluation, is the kind of work a dedicated team can cover. Whoever builds it, ask for a written cost estimate using the components above, an evaluation set you own, and documentation of how permissions map into retrieval.

FAQs

How much does a RAG pipeline cost per month?

In our illustrative model, about $1,350 a month for 100,000 questions, $6,550 for 500,000, and $28,350 for 2 million, infrastructure only. Engineering and evaluation come on top.

What is the most expensive part of RAG?

LLM generation: 78-84% of infrastructure cost in every scenario we modeled, because each answer sends thousands of tokens of retrieved text to the model.

How much does it cost to embed documents?

Very little. At $0.02 per million tokens, embedding 5 billion tokens costs about $115 once. Parsing messy documents usually costs more than the embeddings.

How much does a vector database cost?

For small and mid-sized indexes, often just the plan minimum, such as Pinecone's $50 a month. On read-unit pricing, query cost grows with index size, so very la