ProjectBuild a Permission-Aware Knowledge Hub as an MCP Server: the explanation · Module 8: Indexing and embeddings · part 10 of 22
Part 10 · Module 8: Indexing and embeddings
Module 8: Indexing and embeddings
25 min read·
About 50 minutes · Needs: Module 5 (the chunks table), Module 6 (the seed company) · Code: server/wayfinder/index/loader.py, server/wayfinder/index/chunker.py, server/wayfinder/index/embedder.py, server/wayfinder/index/indexer.py, server/wayfinder/cli.py
Where this fits
This module (blue) is the indexing path: the seed is cut into chunks, the embedding model (amber) turns each chunk into 384 numbers, and both land in PostgreSQL.
Search can only find what was indexed, and only as well as it was cut. This module follows Tidewater Freight's 403 sources from files on disk to 1,154 rows in the chunks table, each with a vector of 384 numbers and a keyword index. It explains what an embedding is, why Wayfinder uses a small model on CPU, how sources are cut into chunks, and how a reindex re-embeds only what changed.
The rest of this project is yours to keep
This project is bought on its own, once, and stays readable afterwards, including the parts added to it later.