Choosing a Vector Database
Compare the practical decisions behind selecting storage for embeddings and semantic search.
Start with workload facts
A vector database is infrastructure, so select it from workload requirements rather than popularity. Estimate the number of documents and chunks, embedding dimensions, update frequency, query volume, expected top-k, metadata filters, geographic needs, and retention period. Include ingestion bursts and rebuilds, not only average traffic. Determine whether users need exact keyword matches, faceting, transactions, or joins with business records. A system that handles a benchmark well may still be a poor fit if it cannot enforce tenant filters, restore a backup, or explain an operational failure.
The storage model
Vector systems store an embedding alongside the text or a reference to text and its metadata. Keeping full text in the index makes retrieval simpler, while storing it in a source database can improve normalization and access control. Decide which system is authoritative and how updates flow between them. Every record needs a stable ID and a document version. Deletions must remove both vector and content references. If the index becomes stale, the application should detect that condition rather than quietly answering from an obsolete copy.
-- PostgreSQL with pgvector: keep tenant filtering in the query.
SELECT chunk_id, document_id, content,
1 - (embedding <=> $1::vector) AS similarity
FROM document_chunks
WHERE tenant_id = $2
AND effective_at <= CURRENT_DATE
AND (expires_at IS NULL OR expires_at > CURRENT_DATE)
ORDER BY embedding <=> $1::vector
LIMIT 5;Managed or self-hosted
Managed services can provide scaling, backups, upgrades, and operational dashboards with less engineering effort. Their trade-offs include recurring cost, vendor-specific APIs, data residency questions, and limits on customization. Self-hosted databases can give more control and predictable deployment boundaries, but the team owns capacity, patching, replication, backups, restore tests, and incident response. Compare the total operating work for your team, not just the service price. A small product may reasonably start with vector support in an existing database if its filtering and scale needs are modest.
Search capabilities to compare
Evaluate approximate nearest-neighbor indexing, exact search options, metadata filtering, hybrid keyword search, namespaces or tenant isolation, bulk ingestion, partial updates, deletes, and reranking integration. Check whether filters are applied efficiently before candidate selection or only after retrieval. Confirm support for your embedding dimensions and distance metric. Also examine pagination, result consistency, SDK quality, observability, and migration tools. A feature list is not enough: run representative queries with real metadata and measure relevance, latency, index build time, and behavior during updates.
Reliability and recovery
RAG infrastructure needs ordinary database discipline. Define availability targets, timeouts, retry behavior, backup frequency, recovery-point objectives, and recovery-time objectives. Test restoring an index and reconnecting it to the source-of-truth documents. Plan for a provider outage: an application may show a limited search result, ask the user to retry, or route to a safe fallback. Do not retry every failure indiscriminately, because repeated indexing or query attempts can amplify an outage and create duplicate work. Make recovery procedures executable and review them before launch.
Security and isolation
Treat embeddings, chunk text, metadata, and query logs as potentially sensitive. Use network controls, encryption, least-privilege credentials, tenant-aware filters, and separate environments. Verify that backups, replicas, analytics exports, and debug traces receive the same protection as the primary index. Be careful with multi-tenant namespaces: isolation must be enforced by trusted application logic or database policy, not by a model-generated filter. Test cross-tenant queries explicitly, including malformed identifiers and empty filters, because a missing filter can become a data disclosure.
Cost and performance
Vector search cost includes embedding generation, storage, index memory, query compute, reranking, network transfer, and rebuilds. Measure p50 and tail latency under realistic concurrency, including cold starts and large metadata filters. Cache stable embeddings and repeated safe searches where appropriate, but avoid caching responses across users or permission scopes. Limit retrieved context so model-token cost does not grow unnoticed. A slightly slower index that retrieves better evidence may reduce total cost by lowering retries, escalations, and incorrect downstream actions.
A sensible decision process
Prototype the same ingestion and query interface against two plausible choices, using a representative dataset and evaluation questions. Record relevance, latency, cost, filtering correctness, update behavior, backup and restore effort, and migration difficulty. Hide provider-specific details behind a repository interface so the application owns document IDs, metadata, and evaluation traces. Choose the simplest option that satisfies the current workload, then set thresholds that trigger a review as volume, tenant count, or availability requirements grow. Infrastructure should support the product’s needs without becoming the product.