Module 0: Project overview: a guided tour of all three parts

Start here · About 30 minutes · Needs: nothing · Code: none (screenshots of the real apps, running locally with Docker on the author's Mac)

Most AI projects stop when the answer looks correct. This project starts there. You will build three things. Part 1 is a video search that sounds reasonable on paper: frames go to one model, speech to two others, and the results are merged by rank. Its answers feel almost correct. Part 2 rebuilds it around one decision, and a whole category of wrong answers disappears. Part 3 is StyleMe, a fashion shopping assistant that decides what the shopper wants before it searches, then proves its behaviour with tests, evals, traces and a deployment.

This page is a tour of the finished work, told through screenshots of the real apps. The 24 modules after it take every piece apart.

Where this fits

Module 0 touches every box on the map, from the outside. The same map opens every module, with that module's parts highlighted.

What it does

The course is one argument in three steps. First, a design that looks right fails on a designed test, and the failure is not a prompt or a chunk size but the architecture: picture and speech live in two separate vector spaces and never meet. Second, embedding what was shown and what was said together, as one vector per moment, removes that failure. Third, a real product needs more than good retrieval: it must decide whether to search at all, ask when it does not know enough, stop asking after three questions, and refuse to describe products it did not find.

The core message runs through all three: an answer that looks right is not evidence; a designed test is.

At a glance

Use cases and examples

A test that can fail. A broadcaster searching its archive asks "which minister spoke in front of the flooded bridge?"; a test with only spoken names would never reveal that the picture is ignored. An e-learning site checks that "the formula on the slide" questions really hit the slide. A hospital training library tests a whiteboard-only answer.

Joint embedding. A newsroom wants "the clip where the reporter says 'record high' over the stock chart". An insurer reviews dashcam footage where the driver's words matter next to what the camera saw. A school indexes lab demos where the teacher names a chemical while pointing at it.

Intent before retrieval. An online shop greets a returning customer without dumping products. A bank's assistant asks one question ("personal or business account?") before searching policies. A SaaS help bot answers "thanks!" with words, not with ten help articles.

When not to bother. A single search box over product titles, a video archive where every answer is spoken, or a prototype nobody will rely on: there, a plain keyword or text search is cheaper and good enough.

How it works

The tour follows the order of the course. The video screenshots were retaken on 28 October 2026, after the security review; the StyleMe screenshots were taken before it. The video screenshots come from separate runs from the comparison report, with the same settings, so their wording and timings differ a little from the report. Where this page quotes an answer word for word, it comes from the report of one run on 27 October 2026 (UTC stamp 20261027T192400Z) with a free-tier Gemini key.

1. The test video and its questions

Every answer in the test video is placed on purpose. The company is fictional: Tidewater Freight, a shipment-tracking architecture review presented by Priya Nair, platform lead. Every picture is drawn by a script and the narration is a synthetic voice, so there is no third-party media.

Three questions carry the tour. v01 asks which meeting room the review is held in; the answer, Harbor-4, is only on the title slide. s02 asks how long raw GPS pings are kept; "45 days" is only spoken. c01 asks which service Tidewater removed in March. The slide shows a service map with one box in red; the speaker says "the service in the red box, that's the one we removed in March". Only the picture holds the name and only the speech says "removed". A cross-modal question needs both.

2. Part 1: the stitched lab

Open the full-size Part 1 architecture diagram

The Part 1 lab, "Video RAG: stitched models". The sidebar reports the API status as ok, two collections (p1_visual, 512 d, and p1_text, 384 d), fusion rrf(k=60), generation with gemini-3.5-flash-lite, and three loaded models: CLIP ViT-B/32, Whisper base and all-MiniLM-L6-v2. "How it works" lists four steps: a frame every 5 s to a CLIP vector, audio to Whisper to about 64-word chunks to MiniLM, the query embedded twice for two searches, and the two ranked lists merged by rank, then answered. The main area takes an mp4, mov or webm of up to 200 MB.

The test video after "Index this video". The title slide reads "Tidewater Freight, Shipment-Tracking Architecture Review, Platform team, Room Harbor-4". The counters show 40 frames (CLIP), 12 text chunks (MiniLM), a length of 3:22 and the video id 3f4b577f016dc1a0, the start of the file's SHA-256. "This exact file was already indexed; nothing re-embedded", so processing took 0.1 s. The report's first ingest of the same file took 18.0 s.

3. Part 1 answers c01: almost correct

Asked "Which service did Tidewater remove in March?", Part 1 answers: "Tidewater removed the service shown in the red box on the service map in March because carrier portals were already performing the same job, which also helped cut a deploy pipeline and late-night pages [0:18-0:54]." The reason is right, the time is right, the name is missing. Embedding took 49 ms, the two searches and fusion 51 ms, generation 1123 ms, 1223 ms in all. Source #1 is the opening transcript chunk, 0:00 to 0:23, with an RRF score of 0.0164.

The five fused sources. Two are transcript chunks ranked by MiniLM, two are frames of the cloud-cost donut chart (2:55 and 3:05, CLIP cosine 0.263) and one more chunk at 1:22. Source #3, 0:18 to 0:54, holds the speech: "now look at the service in the red box, that's the one we removed in March". No frame of the service map made the top five.

This is the binding problem. The speech says "red box"; the name sits inside a picture. Part 1 ranks frames and transcript chunks in separate spaces and merges only their ranks, so nothing ties the words to the box. Even when a frame does rank first, the generator receives it only as a timestamp, which is why Part 1 answered none of the 7 visual questions in the report (the binding problem is the subject of Module 6, "almost correct").

4. Part 1 cannot name the room

v01, "Which meeting room is the architecture review held in?". Part 1: "I could not find the specific meeting room name in the video, though the speaker mentions that the next review will be held in the same room [3:07-3:21]." Source #1 is a frame of the pizza workshop slide at 2:05, CLIP cosine 0.254; source #2 is the closing transcript chunk. The answer is honest, and useless: the room name is printed on the first slide.

5. Part 2: one embedding per moment

Open the full-size Part 2 architecture diagram

The Part 2 lab, "Video RAG: unified embedding", with the same file and the same video id. The sidebar shows one collection, p2_unified (1536 dims), embedding with gemini-embedding-2, upload mode auto, and the steps: ffmpeg cuts about 15 s windows with 3 s overlap, each window becomes ONE Gemini embedding of picture and speech together, one Qdrant collection, one query embedding and one search. Its last line: "No CLIP, no Whisper, no MiniLM, no rank fusion." The counters read 17 chunks.

6. Part 2 answers both

The same c01 question. Part 2: "Tidewater removed the Label Printer Service in March because the carrier portals were already performing the same job [0:12-0:33]." Embedding the question took 833 ms, the search 8 ms, generation 865 ms, 1706 ms in all. Source #1 is the window 0:24 to 0:39 with a score of 0.4778 and a thumbnail of the service map with the red box.

v01 in Part 2: "The architecture review is held in Room Harbor-4 [0:00-0:15]." 1594 ms in all, 6 ms of it the search. Source #1 is the service map window 0:12 to 0:27, which overlaps the end of the title slide; the title slide window itself, 0:00 to 0:15, is source #2.

The search itself takes milliseconds; the time goes to model calls. Part 2 embeds the question through an API, not a local model, so each question is slower.

7. Part 2's sources, and a note that is wrong

Each source carries a note that Gemini wrote at ingest, describing what was on screen and what was said. Source #2, 0:12 to 0:27, lists "Label Printer Service (in red box)". Source #1, 0:24 to 0:39, says the opposite: "Tracking Service (highlighted with a red box in the video, though speaker refers to it as the Label Printer Service)".

Source #1's note is wrong, and it is still in the index. In the first screenshot run the mistake reached the answer ("the Label Printer Service (also referred to as the Tracking Service)"); in the retaken run it did not. Same index, same note, different generation. Notes are a strength of Part 2 and a risk (ingest notes are covered in Module 9, grounded answers).

8. The comparison in numbers

The harness in evals/video/ asked all 26 questions of both running APIs, top 5 sources each.

Read the answer rows with care. Part 2's generator sees Gemini's ingest notes; Part 1's never sees a frame. So the jump from 38% to 96% mixes two changes, the joint embedding and a model describing the frames. The retrieval rows, hit@1 and MRR, are the fair comparison of the two designs. hit@5 is nearly saturated on a video this short (92% against 96%), and this is one run on one video with a free-tier key (Module 10 measures and qualifies every number).

9. StyleMe: the empty chat

Open the full-size StyleMe architecture diagram

"What are we shopping for today?" The promise is in the subtitle: "If I need more detail I'll ask, at most three quick questions, then find matching pieces." Four starters ("A light kurta set for a summer wedding", "Formal shirt and blazer for office, under ₹2,000", "Comfortable sandals for women", "I need something for a party"), a dashed "Search with a photo" button, and a composer that accepts a pasted or dropped PNG, JPG or WebP of up to 5 MB. Top right: How it works and New chat.

10. A greeting, then a question

"hi" gets a greeting and no products: "Hello! Welcome to StyleMe, your personal fashion assistant. What are you looking to wear or buy today?" Then "I need something for a function" gets a badge, "Question 1 of 3", the question "Could you please tell me what kind of function or event you are attending?" and five chips: Wedding, Party, Mehendi, Sangeet and a dashed "Just show me options".

This is the project's name in action: intent over keywords. A keyword engine would search for "function". The planner sees a request too vague to search well, so it asks once and offers answers the shopper can tap.

11. An answer, then results

"It's a wedding" ends the questions. The reply names one product: "For a wedding, the LOUIS PHILIPPE Suit Checkered Men Suit at ₹9,749 is a sharp and elegant choice that will make you stand out." The group "Ethnic wear", searched for "wedding wear outfit for men and women", holds that suit (49%), DENNIS LINGO Men Cargos at ₹949 and TORRY Lace Up For Men at ₹499 (both 48%).

The group title is the category the planner gave its query, not a label on each product, which is why cargos sit under "Ethnic wear". The percentages are match scores, and around 50% they are weak: the catalogue has 203 rows, and not every request has a good answer in it.

12. A detailed request, and Why these?

"cotton kurta for a summer haldi ceremony, women, under ₹1500" needs no question. The planned query is "women yellow cotton kurta for haldi ceremony under 1500": the planner added "yellow", which the shopper never typed. Results: MOKSHI Women Kurta and Palazzo Set Cotton Blend, ₹569 (51%), SKY FAB Women Maxi Yellow Dress, ₹439, and AARTI FASHION Women Printed Rayon Anarkali Kurta, ₹423 (both 49%). The reply begins "Here are some cheerful yellow options", but the set in the picture is pink and the dress orange.

Under the results, "Why these?" opens the turn's reasoning. Decision: search, "enough detail, so it planned the queries below and searched the catalogue". Questions asked: 0 of 3 allowed. One planned query, in the category Ethnic wear. Timings: planner 983 ms, search 771 ms (embedding 764 ms, vector search 7 ms), answer 1.1 s, total 2.8 s. Grounding check: "Passed: the reply names only products, brands, prices and item types that were retrieved."

The grounding check passed although the colours were wrong, because at the time it checked ids, brands, prices and product types, not colours. That gap was found from this screenshot and closed afterwards (see What can go wrong).

13. Search with a photo

A photo with no text, labelled "Search by photo". An image never gets small talk or a question, so the turn goes straight to search, and the group is titled "Similar to your photo". The first card, SURHI Women Kurta and Sharara Set Rayon at ₹439, shows the same picture the shopper sent, at 75%. Then SHINOY Women Printed Rayon Straight Kurta, ₹259 (72%), and MOKSHI's kurta and palazzo set (66%). The reply names the first two with their prices.

Text and photos share one embedding space here too: each catalogue product is indexed as one vector from its image and its title together, the same idea as Part 2 applied to a shop.

14. Vague answers, and the cap

A second chat. "I need something for a function" gets Question 1 of 3, this time without chips. "not sure yet" gets Question 2 of 3, "Are you looking for something traditional like ethnic wear, or more western and casual?", with Ethnic wear, Western wear, Indo-western and "Just show me options".

"something nice I guess" gets results instead of a third question. The query is "stylish party wear outfit"; the reply names the Fnock Fashion Midi/Knee Length Party Dress (₹265) and the Fashion Dream Girls Calf Length Party Dress (₹539). The third card is KIDZAREA Boys Casual Shirt, Waistcoat and Pant Set, ₹449. All three are children's clothes, at 50% to 51%.

The cap is a rule in code, not a sentence in the prompt: after three questions the server forces a search. Here the planner chose to search after two. The results show the cost of vagueness: nobody said who the outfit was for.

15. How StyleMe explains itself

The How it works drawer. "Keyword search matches words. StyleMe works out intent first." Three steps: the planner "decides first" and picks greeting, question or search, asking at most 3 questions per chat; search turns each planned query and the photo into one multimodal embedding; the answer covers "only what it found", and a code check removes a brand, price or item type that was not retrieved. Below: language model gemini-3.5-flash-lite, embeddings gemini-embedding-2, tracing off until Langfuse is configured.

16. On a phone

The phone layout. "I need something for a party" gets Question 1 of 3, "Are you looking for something ethnic or western?", with the chips wrapping onto three rows. The header keeps Shop, About and two round buttons (How it works, New chat); the composer placeholder shortens to "What are you looking for?".

Results in two columns: VASTRANIKHAR Women Ethnic Dress Gold, Pink at ₹396 and FABFAIRY Women A-line Green Dress at ₹439 (both 50%), then MOKSHI's set at ₹569 (49%), long titles cut with an ellipsis, and "Why these?" below.

17. StyleMe, measured

The evals run the real HTTP app over 46 hand-labelled conversations and two retrieval sets. Every gate passes except latency: the search-turn p95 of 39.6 s matches the free tier's 429 backoff, while the median is 4.2 s. That eval run indexed all 203 rows; the current code indexes 202 (product 4 is excluded, see below).

Code tour

Every later module has its own code tour. Here is where things live.

Parts 1 and 2 expose the same API (POST /ingest, POST /search), so one harness can measure both.

What happens when you ask c01 in both parts

1. You type "Which service did Tidewater remove in March?" in the Part 1 lab, with top k 5 and "only last indexed

   video" ticked.

2. Part 1 embeds the question twice: once with CLIP's text tower (512 d), once with MiniLM (384 d).
3. It searches p1_visual and p1_text, each for top k times 2 = 10 candidates.
4. Reciprocal rank fusion with k = 60 merges the two lists by rank. In the screenshot, the opening transcript chunk

   wins with 0.0164.

5. The generator gets transcript text for chunks and only timestamps for frames, and writes "the service shown in the

   red box".

6. In Part 2, the same question becomes one RETRIEVAL_QUERY embedding (1536 d) and one search of p2_unified.
7. The top window, 0:24 to 0:39, comes back with its ingest note, and the generator writes "the Label Printer

   Service", citing times.

Design decisions

When to choose differently

What can go wrong

Answers that look right. Part 1's c01 answer has the right reason and time; only a test for the name catches it.

Notes that are wrong. Part 2 stores a Gemini note per window, and the note for 0:24 to 0:39 names the wrong service. A clean answer in one run does not mean the index is clean.

Mixed causes. Part 2's answer rate rises partly because of its notes. Claim the redesign's win on retrieval metrics, not on answer rates alone.

Fixed: a reply that described colours it did not have. The haldi reply called a pink set and an orange dress "cheerful yellow options", and the grounding check passed it. Found in the screenshot pass. A fifth check now drops any sentence stating a colour that no backing product title has, or falls back to a template; the reply is a test case in tests/test_grounding_colours.py. The check is strict: an honest "we have no yellow ones" is dropped too.

Fixed: a product whose photo was a necklace. Product 4 is listed as "Regular Fit Women Black Cotton Blend Trousers", but its image is a gold chain necklace. Found in the screenshot pass and confirmed by auditing all 203 images; it is now in excluded_products.csv, so 202 products are indexed.

Fixed: results grouped under "General". Also from the screenshot pass: groups are never titled "General", a photo-only search is titled "Similar to your photo", and cards are sorted by score. Each fix is pinned in the frontend's ux.test.tsx.

Slow tails on the free tier. StyleMe's latency gate fails at p95 because of 429 backoff. A key with a higher quota or client-side pacing is the fix; the 9 s threshold stays.

Try it yourself

Everything runs locally with Docker. Copy the example settings and add your own Gemini key (Part 1's search works without one, but its answers need it):

cd source-code
cp .env.example .env
make up-video-compare

What you should see

Part 1's lab on http://localhost:8511 and Part 2's on http://localhost:8512 (the first Part 1 build is large: about 5.7 GB). Upload samples/test_video.mp4 in each and ask c01, v01 and s02. make eval-video then runs all 26 questions through both APIs and writes a report; it calls Gemini, so it uses your quota.

make up-styleme
make ingest

What you should see

StyleMe on http://localhost:8097. The ingest embeds 202 products and is safe to run again. Type "hi", then "I need something for a function", and open "Why this reply?".

make test

What you should see

Every test suite runs on fakes, with no network, keys or models.

Check yourself

1. Part 1 ranked the right moment first for c01, yet the answer failed. Why?

Its sources were scored in two separate spaces and fused by rank, and frames reach the generator only as timestamps. Nothing tied the spoken "red box" to the name drawn in it.

1. Part 2's answers rose from 38% to 96%. Why is that not a clean measure of the joint embedding?

Part 2's generator also reads ingest notes that Gemini wrote while watching each window. Retrieval metrics (hit@1 58% to 85%, MRR 0.74 to 0.89) compare the designs fairly.

1. Why did "hi" not return any products in StyleMe?

The planner decides intent before retrieval. A greeting ends the turn with a reply and never reaches the search.

1. The haldi reply passed the grounding check although its colours were wrong. What changed?

The check then covered ids, brands, prices and product types only. A colour check was added that drops sentences stating a colour no backing product has.

Key terms

Where to go next