Creating Kb

creating-kb

Turn a pile of files into a portable, deployable knowledgebase. The output

is a .skill bundle — an ordinary zip — containing a BM25 inverted index, the

chunk text, a pure-Node searcher, and a query protocol. It has no embedding

model and no semantic search: retrieval is lexical, and the consuming agent

supplies the semantic layer by expanding the query at search time. That is what

makes the bundle portable — any agent that can run node can query it with no

npm install, no model download, and no network.

The whole toolchain is JavaScript so one implementation serves both this builder

and the in-browser packer. Build with the bundled script; do not hand-roll the

index.

SCRIPTS=/mnt/skills/user/creating-kb/scripts
node $SCRIPTS/build_lexkb.js CORPUS_DIR --out /tmp/kb --name my-kb --zip

Workflow

1. Gather the sources

Collect the files into one directory. In a Claude.ai chat, uploads land in

/mnt/user-data/uploads/ — point the builder there. Otherwise use any path the

user names. Supported extensions default to txt,md,html,htm; pass --ext to

change them.

This MVP interface is bounded by how many files a chat can accept. For a large

corpus, stage the files in a directory first, or use the browser packer (built

from the same scripts) that runs entirely client-side.

2. Build the bundle

SCRIPTS=/mnt/skills/user/creating-kb/scripts
node $SCRIPTS/build_lexkb.js /mnt/user-data/uploads \
  --out /tmp/kb --name my-kb --zip \
  --source "human description of the corpus"

The script chunks each file, builds the BM25 index, writes the bundle dir

(SKILL.md + search.js + index.json + chunks.jsonl), and — with --zip —

emits my-kb.skill next to --out.

3. Deliver

Move the .skill to the outputs directory and give the user a download link:

cp /tmp/my-kb.skill /mnt/user-data/outputs/

[Download my-kb.skill](computer:///mnt/user-data/outputs/my-kb.skill)

Tell the user how to deploy it: unzip into an agent's skill directory (or upload

it as a skill). The bundle's own SKILL.md then drives querying — the consuming

agent reads it, expands each question into search terms, and runs the bundled

search.js. No further setup.

Choosing chunk size

The retrieval unit and the reasoning unit are decoupled, which makes chunk size a

low-stakes choice. search.py/search.js rank on the whole chunk (best

recall) but return only the query-densest passage of it by default

(--snippet, ~1200 chars), so a big chunk does not flood the consuming agent's

context with surrounding noise. Index for recall; the searcher handles signal.

--target-chars controls chunk size (whole paragraphs are packed up to the

target; --target-chars 0 makes each file one chunk). Lexical BM25 tolerates —

and on a real-corpus sweep slightly *preferred* — larger chunks than

embedding-based retrieval, because there is no vector to dilute: BM25 scores

individual term presence with length normalization, so a big chunk still ranks on

the exact terms it contains.

• Default: --target-chars 0 (whole document). Best recall, fewest chunks;

the snippet return keeps reasoning context focused.

• Long, multi-topic files where you want tighter citation units: 1500–4000.
• 500 only if you need very fine-grained chunk ids and accept more chunks.

Verifying the bundle

Test before delivering. Run a query against the freshly built bundle and confirm

it returns sensible hits:

node /tmp/kb/search.js --query "a representative question" \
  --core "key term" --expand "synonym" --k 3

Each hit's text is the query-focused passage by default; add --snippet 0 to

inspect a full chunk.

search.js prints JSON {"hits": [...]}. Confirm the right chunks surface.

What ships in the bundle

| File | Role |

|---|---|

| SKILL.md | the query protocol the consuming agent follows (expand → search → cite) |

| search.js / search.py | equivalent BM25 + RM3 + metadata-filter searchers; return query-focused passages (matched sentences kept in neighbour context, merged); the agent runs whichever runtime it has |

| index.json | precomputed inverted index (postings, df, doc lengths, BM25 params) |

| chunks.jsonl | chunk text + structured metadata |

Both searchers are thin readers of the same neutral JSON index, so the bundle

runs in a Node-only or a Python-only consumer. Metadata stays structured (not

folded into the indexed text), which lets the consuming agent filter on it

(--filter section=blog, --filter date>=2025).

Scripts

• scripts/build_lexkb.js — chunker + BM25 index builder + .skill writer.
• scripts/search.js — the JS runtime searcher, copied verbatim into every

bundle. It owns the tokenizer; the builder imports it so index and queries

tokenize identically.

• scripts/search.py — the Python runtime searcher, copied verbatim into every

bundle; a thin reader of the same neutral JSON index, parity-pinned to

search.js (identical results on a shared index).

• scripts/zipstore.js — pure-JS ZIP-STORED writer (used by the builder; shared

with the in-browser packer).

• scripts/bundle_SKILL.md — the query-side SKILL.md template written into each

bundle.