Local vector search for coding agents: From 507 MB to a 7-line loop

Paul Irish

Published: November 12, 2026

This week we launched Modern Web Guidance 1.0, an open-source skill that helps coding agents use modern web platform features instead of legacy workarounds. This post takes a look at the local search engine that finds the most relevant guide for your task, without wasting your tokens.

If you tell an engineer that you built a knowledge base with 1.7 MB of Markdown text for their coding agent, they will ask two questions:

1. "How many tokens will this dump into my context window?"
2. "What kind of bloated vector database does it require?"

Over the last six months, we rebuilt our local retrieval engine three times. We went from 507 MB of C++ bindings, then down to 23 MB of pure JavaScript using TensorFlow.js, and then down to a 7 MB package where query embedding is a 7-line for loop, all while improving on accuracy.

Agentic RAG and progressive disclosure

Dumping 1.7 MB of Markdown text containing over 200 HTML/CSS/Web API guides into AGENTS.md isn't practical. We believe in progressive disclosure, but know that a Markdown index will stop being token-efficient at scale.

To solve this, we use agentic RAG backed by a search CLI:

1. At rest, the agent only sees the skill's description (~205 tokens). The description follows Anthropic's skill trigger guidelines so that the agent invokes the skill on relevant frontend tasks.
2. On frontend tasks, the agent can use the CLI to search: modern-web-guidance search "animate height auto", getting 10 relevant matches (~530 tokens).
3. The agent runs modern-web-guidance retrieve animate-to-intrinsic-sizes to pull the entire guide (about 800 tokens). It's now equipped with the implementation best practices for compatibility, fallbacks, and technique.

Why a CLI skill instead of an MCP server or a web search?

• MCP tools either waste tokens or have unreliable trigger mechanisms (due to dynamic tool discovery). Our skill's description is minmaxed for effectiveness and enables the agent to consume guidance only when appropriate.
• Built-in web search surfaces 2015 Stack Overflow answers and reference docs rather than up-to-date, curated DO / DO NOT implementation patterns. It also takes seconds over the network and fails in offline sandboxes, whereas the local CLI runs on CPU in milliseconds.

Gen 1: The 507 MB C++ prototype

The first version of Modern Web Guidance used a local RAG stack: LanceDB, Apache Arrow, and Transformers.js with onnxruntime-native.

It worked fine on a developer machine, but it was a rough fit for an agent skill:

• When you add a skill to a project (or an agent invokes it with npx), you don't expect to wait on an npm install downloading architecture-dependent .node binaries or node-gyp C++ compilation that routinely fails in cloud sandboxes.
• The extracted node_modules directory weighed 507 MB.

Gen 2: Golfing TensorFlow.js down to 23 MB

We wanted zero native compilation and instant execution anywhere Node runs. So in April, we replaced the native stack with a pure-JavaScript CPU engine running all-MiniLM-L6-v2 on @tensorflow/tfjs:

• Shimmed out onnxruntime-node in esbuild and replaced the Hugging Face tokenizer with a tiny in-tree BERT WordPiece tokenizer (0 token-ID mismatches).
• After some source-map exploring, pruned @tensorflow/tfjs-backend-cpu's 400+ modules down to the 26 kernels MiniLM actually uses, shrinking the JS engine from 6 MB to 830 KB.
• Dropped LanceDB for build-time gzipped vectors and an in-memory <1 ms dot-product scan across all 200 or so guides.
• Put the embedder behind an import() call so retrieve and list finish in less than ten milliseconds.

That got the npm package down to 23 MB with no runtime dependencies. We shipped it, and agents ran about 750,000 searches against it.

Where MiniLM broke in production

A couple months in, we looked at trajectories where agents had the skill installed but still wrote legacy code.

In one Codex run, a dev asked for a delete confirmation prompt. We have a whole guide on building accessible confirmation modals with HTML <dialog closedby="any">.

The agent did the right thing and searched:

npx modern-web-guidance@latest search "confirm destructive action"

The CLI returned []. So the agent shrugged and wrote window.confirm().

Why did it return empty? When we dug into the logs and embeddings, two retrieval gaps jumped out:

1. all-MiniLM-L6-v2 was trained around 2018–2020, so its WordPiece vocabulary literally does not know post-2020 web APIs exist. When an agent searches for fetchpriority, popover, or startViewTransition, the 2018 tokenizer shreds fetchpriority into four meaningless subwords: fetch · ##pr · ##ior · ##ity.
2. To prevent out-of-scope queries from returning random guides, an earlier PR had added a strict 0.38 cosine similarity cutoff. Because MiniLM couldn't bridge the 3-word phrase "confirm destructive action" to "dialog" above 0.38 (scoring 0.349, while "popover anchor" got 0.362 and "autofill address form" got 0.354), the gate threw valid matches away. Worse, whenever search returned [], our skill instructions told the agent to fall back to modern-web-guidance list, dumping all ~200 guide descriptions (~1,500 wasted tokens) into context.

Fixing retrieval: real telemetry, hybrid search, and returning 10 results

Our original benchmark was 850 synthetic queries generated from the guides themselves, which matched the guide phrasing too closely to catch these failures. So we built a new evaluation harness (pnpm benchmark:rag) around 4,000 live searches sampled from opt-out CLI telemetry (which logs only the agent's 2–5 word search argument, never user prompts or repository code, and respects DISABLE_TELEMETRY=1) plus 1,029 searches mined from nightly agent evals across Claude Code, Codex CLI, and Gemini CLI.

Replaying our git history against those held-out queries showed that the 0.38 cutoff had spiked empty [] responses 8x. With the benchmark in place, we landed two fast improvements before touching the embedding model:

1. Hybrid keyword + semantic search: Dense vectors blur exact API names (:user-invalid, showModal, details-name), while keyword search misses intent queries when the agent doesn't know the API name. We added a zero-dependency BM25 (if you know, you know) keyword index with plural stemming, indexed each guide's web-feature-ids, and fused BM25 with semantic ranking. Now search only returns [] when both semantic similarity is <0.30 and BM25 has zero keyword hits.
2. 10 compact results instead of 5 verbose ones: The skill instructions already told agents to ignore similarity and tokenCount. By stripping those unused fields, we fit 10 compact {id, description} results into the exact same ~530-token budget as 5 verbose ones:

// Before: 5 verbose results (~105 tokens each = ~525 tokens)
{
  "id": "dialog",
  "category": "html",
  "description": "Build accessible modal and non-modal dialogs with native <dialog>, showModal(), and closedby.",
  "featuresUsed": ["dialog", "dialog-closedby"],
  "tokenCount": 780,
  "similarity": 0.49
}

// After: 10 compact results (~53 tokens each = ~530 tokens)
{"id":"dialog","description":"Build accessible modal and non-modal dialogs with native <dialog>, showModal(), and closedby."}

Together, these two changes lifted our search hit rate on held-out telemetry queries from 80.6% to 94.7%, measured as Hit@K (whether the expected guide appears in the top K results).

Gen 3: Pre-baked embeddings and a 7-line for loop

Even at a 94.7% hit rate, our telemetry showed a glaring mismatch: 86.7% of live agent searches were just 2 to 5 words long (median: 3 words). Every time an agent searched a 3-word phrase, Node was booting TensorFlow.js, parsing a 22.6 MB neural network, and burning 770 ms of cold-start latency and 346 MB of peak RAM.

When asked if there was a lighter way to embed 3-word queries without a neural net, our Antigravity agent came back with a stack of unfamiliar ML distillation techniques: Model2Vec, Tokenlearn, Matryoshka dimensions, Adagrad. (If you want the nerdy details, there's more on our search quality page.)

Standard local RAG forces both documents and queries through the same small model on the user's CPU. But our ~200 guides don't change after we cut an npm release, so there's no reason to embed them with a tiny 2018 model. Instead, embed-guide-chunks.ts pre-bakes all of our guide vectors at release time in the cloud using gemini-embedding-001 and ships them as binary 256-dimensional Int8 vectors (guide-chunks.i8, 335 KB gzipped).

How does an offline, zero-API-key CLI project a user's live query into that exact same Gemini vector space? Because agent queries are almost always 2 to 5 words, we don't need a transformer at runtime. We just need a lookup table of vectors for known words (table) and a smart way to sum them. At runtime, the entire 23 MB TensorFlow.js pipeline collapses into a 7-line for loop in static-embedder.ts:

// Each word's weight and Int8 scale are pre-folded into `scales[i]`
for (const [i, f] of enc) {
  const wt = scales[i] * f;
  const offset = i * 256;
  for (let k = 0; k < 256; k++) {
    out[k] += wt * table[offset + k];
  }
}

To build that lookup table (train-query-words.ts), we took a 29,165-word vocabulary (where modern web terms like fetchpriority, light-dismiss, popover, and scrollytelling are standalone whole words), embedded them with gemini-embedding-001, and tuned the word vectors offline against 33,000 real web queries so multi-word combinations land in the right spot (for example, "light" + "dismiss" points toward <dialog closedby="any">, not illumination or color schemes). Compressed to Int8 (query-words.i8), the entire lookup table is 5.3 MB gzipped.

Why not use that static lookup table to embed the guides too, and skip the cloud build step? We tried that first and it fell apart on full paragraphs: summing 3 query words stays sharp, but averaging hundreds of words across a full guide collapses every guide into a narrow cone near the generic English average (to the point where "best pizza in new york" scored a 0.899 cosine similarity against our dialog guide, higher than the query "dialog" itself at 0.844!). Pairing 3-word static query lookups (query-words.i8) against full-model Gemini guide embeddings (guide-chunks.i8) gives us cloud-grade accuracy while queries still run offline in 30 µs.

The scoreboard across all three generations

Across our release milestones, Hit@K rose from 78.2% with whole-document embeddings to 95.4% with the static Int8 embedder (compared to 96.9% with the cloud gemini-embedding-001 API):

Here's how the three generations stack up after landing the static embedder:

And if you run that broken Codex query from earlier:

npx modern-web-guidance@latest search "confirm destructive action"

The dialog guide comes right back at the top of the list.

(You can explore the interactive charts across all milestones, query lengths, and K horizons on our Search Quality Dashboard.)

Code is cheap; measuring is the work

An agent can write a BM25 ranker or a word-vector distillation script in minutes. Generating candidate implementations isn't the bottleneck anymore. Knowing whether a change actually helped (or quietly broke 9% of production searches like our 0.38 gate did) is almost the entire job.

Here's how we approached this system:

• Define the north star goal and invest in the measurement harness over any single implementation. We spent far more time building pnpm benchmark:rag, curating 5,000 real queries, and calibrating our automated grading against human ratings than writing any retriever.
• Build a rubric for how you and your agents judge quality, and expect it to evolve once it hits real-world data. People call this "measurement-first", which is only half right: you can write basic known-answer checks on day one, but you can't design a good rubric in a vacuum before inspecting real failures (Shankar et al. call this criteria drift in Who Validates the Validators?). Seeing agents trigger 1,500-token list fallbacks is what forced us to penalize token bloat alongside recall.
• With trusted measurements in place, experiment aggressively with different implementations and let the scoreboard settle architecture debates. Once a benchmark grades 5,000 queries in seconds, you don't have to argue in PRs about whether a 7-line Int8 loop can beat a 23 MB transformer. Try five approaches in an afternoon and keep the winner.

All of the code (including the embedder, distiller, and benchmark harness) is open source, of course.