Lesson 4: Vector Stores

A vector database is just another Retriever

Module 7 established that a database sits behind a repository interface and business logic never imports a driver. A vector store is not an exception, and treating it as one is how teams end up unable to change it.

class Retriever(Protocol):
    """Finds relevant chunks. Independent of how they are stored."""

    async def search(
        self,
        query_vector: list[float],
        *,
        top_k: int,
        filters: dict[str, object] | None = None,
    ) -> list[ScoredChunk]: ...

    async def upsert(self, chunks: list[EmbeddedChunk]) -> None: ...

    async def delete_by_document(self, document_id: str) -> None: ...

Three notes on the design. delete_by_document exists because deletion is the operation people forget until a document is retracted and its chunks keep being retrieved. filters is part of the interface because Lesson 3 showed that pre-filtering must happen inside the search rather than after it. And the interface takes a vector rather than a query string, so that embedding stays a separate concern and the same retriever works with any embedding model.

class QdrantRetriever:
    def __init__(self, client: QdrantClient, collection: str) -> None:
        self._client = client
        self._collection = collection

    async def search(self, query_vector, *, top_k, filters=None) -> list[ScoredChunk]:
        response = await self._client.search(
            collection_name=self._collection,
            query_vector=query_vector,
            limit=top_k,
            query_filter=to_qdrant_filter(filters),
        )
        return [to_scored_chunk(point) for point in response]

An in-memory implementation for tests is a list and a loop, and it makes retrieval logic testable without running a database, exactly as Module 7's in-memory repository did.

Index types and the recall and latency tradeoff

Searching two million vectors exhaustively is accurate and slow. Approximate nearest neighbour indexes trade a little accuracy for a large speed gain.

Flat, meaning exhaustive search, compares the query against every vector. Perfect recall, linear cost. Correct below roughly a hundred thousand vectors, and a useful ground truth for measuring how much recall your approximate index is losing.

HNSW builds a navigable graph and walks it toward the query. It is the common default: fast, good recall, and memory-hungry because the graph is held in memory. Two parameters matter. Higher ef_construction builds a better graph more slowly. Higher ef_search explores more at query time, improving recall and increasing latency, and it can be raised per query, which lets you spend more on queries that matter.

IVF partitions vectors into clusters and searches only the nearest few. nprobe controls how many clusters are searched, trading recall for speed. It is more memory-efficient than HNSW and generally slightly worse at the same recall.

Quantization compresses vectors, typically from 32-bit floats to 8-bit integers or binary. Storage falls dramatically, recall falls somewhat, and a common pattern is to search the compressed vectors and rescore the top candidates with full precision.

The one thing to take from this. These are all points on the same curve: recall against latency against memory, choose two. The right response to a slow search is not automatically a different index but a decision about which of the three you are willing to spend.

Measure the recall you are losing. Build a flat index over a sample, run your evaluation queries against both, and compare. Teams often discover their approximate index is at 85 percent recall when they assumed 99, and that missing 15 percent is invisible without the comparison.

Keeping the store swappable

The registry pattern from Module 7 applies unchanged.

retriever_registry = ProviderRegistry()
retriever_registry.register("qdrant", lambda s: QdrantRetriever(build_qdrant(s), s.collection))
retriever_registry.register("pgvector", lambda s: PgVectorRetriever(build_engine(s)))
retriever_registry.register("memory", lambda s: InMemoryRetriever())

retriever = retriever_registry.create(settings.retriever_name, settings)

What swapping actually requires, because the interface is necessary and not sufficient. Filter syntax differs between stores, so to_qdrant_filter is not portable and each implementation needs its own translation. Index configuration differs, and the tuning you did for one does not transfer. Hybrid search support differs, and a store without it means running a separate keyword index. And you must rebuild the index in the new store, since the vectors can be exported but the index structure and its tuning cannot.

The interface makes the change a bounded piece of work in known places rather than a rewrite. That is the realistic claim, and it is worth the effort.