Lexical Kb

lexical-kb — query an embedding-free knowledgebase

This KB has no semantic search and no embedding model. Retrieval is pure

lexical BM25 over a precomputed inverted index. That design moves one job onto

you: bridging the gap between how the user phrases a question and how the corpus

phrases the answer. An embedding model would do this with a vector; here **you

are the semantic layer** — you expand the query into terms before searching.

Corpus: {{SOURCE}} ({{CHUNK_COUNT}} chunks).

The retrieval protocol — follow every step

A raw user question fed straight to BM25 underperforms: it matches only the

exact words the user happened to use. The expansion step is what makes lexical

retrieval competitive with embeddings. Do not skip it.

• Read the question. Extract core terms — the essential nouns, proper

nouns, and identifiers the answer MUST contain. These carry full weight.

• Generate expand terms — synonyms, morphological variants (plural/verb

forms), acronym expansions and contractions, and adjacent concepts. These

carry lower weight. This is the work the missing embedding model would have

done. Be generous: 5–15 expansion terms is normal.

• Run the searcher. It ships in this bundle in two equivalent runtimes —

node search.js or python3 search.py, identical flags and identical

results. Use whichever your environment has. Pass the user's original

question via --query AND your term groups — expansion is additive, it

never replaces the user's words:

   node search.js \
     --query "how does centered simhash differ from random projection?" \
     --core "simhash" --core "centered" \
     --expand "random projection" --expand "hyperplane" --expand "LSH" \
     --expand "binary quantization" --expand "hamming distance" \
     --k 5

--core/--expand are repeatable; pass phrases, the searcher tokenizes

them. The --query terms contribute at a low floor weight so a curated

synonym can lift a result but can never drop a doc the literal question would

have matched. Defaults: core 1.0, expand 0.4, query-floor 0.25, top-k 5.

Keep expansion targeted — terms too generic ("system", "process") leak into

unrelated chunks and blur the ranking. A precise word can mislead too if it

is polysemous: prefer the disambiguating phrase as one --core term (e.g.

--core "centered simhash") over a bare ambiguous word (--core "centered",

which also matches "centered around …" in unrelated chunks). The passage

extractor is lexical too, so it will highlight the wrong sense rather than

correct it.

• Read the returned chunks. Answer from them, and cite chunk ids inline.

The chunks are the source of truth the user installed. When a chunk

contradicts your prior knowledge, the chunk wins — say so. When the chunks

do not contain the answer, say that plainly rather than filling the gap from

memory.

When you cannot expand — RM3 fallback

If the query is outside any domain you can expand confidently, pass it raw with

pseudo-relevance feedback. The searcher harvests expansion terms from the

corpus's own top hits — model-free, weaker than your expansion, and prone to

drift when the first pass is off-topic, so prefer real expansion when you can:

node search.js --query "the user's raw question" --rm3 --k 5

Metadata filtering

Each chunk carries structured meta (e.g. title, source_path, section).

Filter on it with --filter (repeatable). Filtering narrows by attribute; it

does not rank — combine it with term search.

node search.js --core "factions" --filter "section=blog" --filter "date>=2025" --k 5

Operators: =, !=, ~ (substring), >, >=, <, <= (numeric when both

sides parse, else lexicographic — ISO dates sort correctly).

Passage vs. full document

Ranking uses the whole chunk (best recall), but each hit's text is by default

the query-densest passage of that chunk (~1200 chars), not the entire chunk —

so your reasoning context is signal, not the surrounding noise of a long document.

Each matched sentence keeps its neighbours (--context, default 1 each side) so

it reads in context rather than as an orphaned fragment, and nearby matches merge

into contiguous passages; ' … ' marks elisions between them. When a hit is a

passage, the result carries full_chars (the chunk's full length). If you need

the complete document for a hit — broader context, a quote in a section the

passage elided — re-run with --snippet 0:

node search.js --query "…" --core "…" --snippet 0 --k 3

Tune with --snippet 2000/600 (budget) and --context 2/0 (neighbours).

Output

The searcher prints JSON: {"hits": [{id, score, text, meta, full_chars?}, ...]},

sorted by descending BM25 score. text is the focused passage (or the full chunk

if it was already short / --snippet 0); full_chars appears only when text

is a passage. Surface the top hits to the user with their ids, then answer using

them as authoritative context.

Mechanics

• Pure stdlib — Node or Python. No npm install / pip install, no model

download, no network.

• The bundle is self-contained: search.js + search.py (pick one),

index.json, chunks.jsonl. Run the searcher from inside the bundle

directory (it defaults --index to its own location) or pass

--index /path/to/bundle.