Vector Embeddings and Chunking
Understand how documents become searchable vectors and how chunk boundaries affect answer quality.
What an embedding represents
An embedding is a numeric representation of text produced by a trained embedding model. Texts with related meaning tend to occupy nearby positions in a high-dimensional space, allowing a search system to compare a question with document passages. Similarity is a useful approximation, not a guarantee that two passages answer the same question. Embedding models differ by language, domain, dimension, and recommended distance metric. Choose one that fits the content and keep the model version with each indexed record, because changing models requires a controlled re-index rather than mixing incompatible vectors.
Chunking is information design
Chunking decides what unit the retriever can return to the model. A chunk that is too small may lose definitions, conditions, or the subject of a sentence. A chunk that is too large may contain several unrelated topics and consume the context budget. Good boundaries often follow headings, paragraphs, list items, or table rows. Keep a small overlap when a concept crosses boundaries, but do not use overlap as a substitute for understanding document structure. Inspect real documents and tune chunking against questions users actually ask.
from openai import OpenAI
import numpy as np
client = OpenAI()
text = open("handbook.txt", encoding="utf-8").read()
chunks = [text[i:i + 800] for i in range(0, len(text), 700)]
vectors = client.embeddings.create(
model="text-embedding-3-small", input=chunks
).data
query = client.embeddings.create(
model="text-embedding-3-small",
input=["Who is eligible for parental leave?"]
).data[0].embedding
scores = [np.dot(query, item.embedding) for item in vectors]
for index in np.argsort(scores)[-3:][::-1]:
print(scores[index], chunks[index])Preserve structure and meaning
During extraction, retain headings, numbering, captions, table labels, and relationships between a section and its content. A paragraph called “Eligibility” is less useful when indexed without the policy title or section heading that gives it meaning. Tables need special handling because reading order can be lost when cells are flattened. You may store both a searchable text form and structured fields for filtering or rendering. Keep the original file reference so a user or reviewer can inspect the source when an answer is disputed.
Metadata makes retrieval safer
Every chunk should carry enough metadata to support filtering, citations, freshness checks, and deletion. Common fields include tenant or user scope, document ID, source, title, section, page, language, document type, publication date, effective date, and ingestion version. Metadata should be generated from trusted system fields where possible, not copied blindly from document text. Apply authorization filters before or during retrieval so unauthorized chunks never enter the model prompt. A secure response cannot reliably undo a disclosure that already happened inside the generation context.
Choosing similarity settings
Vector search commonly compares embeddings with cosine similarity, dot product, or a distance measure supported by the index. The numerical score is meaningful only within the model and index configuration that produced it. Do not assume a score threshold transfers across embedding models or datasets. Start by retrieving a generous candidate set, inspect relevant and irrelevant results, and select a threshold or top-k value using evaluation data. Record the query, model version, index version, filters, candidates, and scores so changes can be compared rather than guessed.
Hybrid search and reranking
Semantic search is strong when a question uses different words from the source, while keyword search is strong for exact names, product codes, legal clauses, and error messages. Hybrid retrieval combines both signals. A reranker can then read a smaller candidate set and reorder it with a richer relevance model. These stages add latency and cost, so use them where they improve measured results. Keep the original candidate scores and final ranks in traces; otherwise it becomes difficult to determine whether a reranker fixed a miss or simply hid a retrieval problem.
Common chunking failures
Blind character splitting can separate a heading from its explanation, cut a procedure between a condition and its exception, or mix footer text into the answer. Excessive overlap can create duplicate evidence and crowd out useful passages. Embedding an entire document may retrieve broad topics but fail to provide the precise supporting detail. Another failure is indexing stale and current versions together without effective dates, which lets the model blend contradictory rules. Review retrieved examples for these patterns and fix the representation before tuning prompts.
A repeatable tuning method
Create a small corpus of representative documents and questions, including exact lookups, paraphrases, multi-part questions, table questions, and questions with no answer. Compare chunk sizes, overlap, metadata filters, similarity search, hybrid search, and reranking one change at a time. Label whether the needed evidence appears in the top results and whether the returned passage is sufficiently complete. Keep the best configuration with its embedding model and index settings. This turns chunking from a one-time guess into a measurable part of the retrieval system.