https://gitlab.synchro.net/main/sbbs/-/commit/2ff8d37d997adb52be2b332c
Modified Files:
ctrl/chat_llm.ini exec/chat_llm.js llm_index.js exec/llm_index/dokuwiki.js
Log Message:
chat_llm: index wiki pages per section and inject the matching passage

Retrieval was selecting the right page and the injector was discarding
the part that answered the question. Wiki pages were indexed as one
document each and then cut to their first 800 characters for the
prompt. On one live corpus, 384 of 541 wiki pages (71%) were longer
than that, mean page length 4932 characters -- so the model typically
saw the first sixth of a page and nothing else.

The failure this produced is worth spelling out, because it looks like
a model problem and is not. Asked whether a semaphore file could
recycle the mail server, retrieval ranked the semaphore-files page
FIRST; its "Recycle Semaphore Files" section begins 3388 characters in
and was cut. What survived was that page's generic example, "touch /sbbs/data/dothing.now". The model then answered with an invented "mail_servers_recycle_now.sem", patterned on the one example it had
been shown, and defended the invention when challenged. Truncation
chose what the model confabulated from.

Five changes:

Wiki pages are split into one chunk per heading. Splitting happens on
the raw text, before markup stripping -- afterwards a heading is indistinguishable from body text. Each chunk leads with a breadcrumb
so a subsection still carries its page's subject, and cites a #anchor
deep link so an answer points at the section it came from.

What survives the per-chunk cap is now the passage matching the query
rather than the head of the document.

Sections of one page are capped in the result set, default 2. A page
used to occupy exactly one slot by construction; per-section chunks
removed that guarantee, and a single long page's sections could fill
the whole window -- one query came back with eight sections of the
same page and none of the page that held the answer.

Tokens are folded to a singular form. There was no stemming at all,
so "recycles" could not match "recycle": the page holding the answer
to one phrasing ranked 76th. Folding is plurals only -- verb-suffix
chopping ("recycling" -> "recycl") does not converge on the form
"recycle" folds to, so it would lose as many matches as it gained.
tokenize() is duplicated between chat_llm.js and llm_index.js and the
two MUST agree, so both copies change together; a short exception list
covers tokens whose trailing s is integral ("news", "windows").

The appended citation follows the retrieved chunk that best overlaps
the reply instead of the top-ranked hit. BM25 ranks against the
question, and once pages are split the highest-scoring section is
frequently not the one the model used. Overlap is scored by Jaccard,
not a raw count, because a long chunk has more distinct tokens and so
more accidental matches.

An unlabeled wiki link now keeps only its page component, so indexed
prose reads "the file ctrl/recycle" rather than "dir:ctrl/recycle" --
the namespace prefix was reaching users as though it were part of the
path.

Note for anyone tuning thresholds afterwards: per-token scores are on
a different scale once chunks get shorter, so a threshold carried over unchanged is markedly less selective than it was. Measured on real
traffic across this change, the median more than doubled and a
volunteer-answer gate went from firing on 15% of turns to 48%.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net