TextGen 5.0 — Custom Build

1.The short version

Four things to say out loud, then the detail is just backup.

1. It's the real app, not a replacement. Same codebase and everything upstream ships — multiple backends, OpenAI/Anthropic-compatible API, tool calling & MCP, vision, file attachments, LoRA training, notebook, themes — running on a newer snapshot than the public v4.9 release.
2. Memory is the headline. Upstream ships no built-in persistent memory for characters; that's left to community extensions with their own dependencies. This build has a purpose-built memory layer with no heavy packages: per-character JSON store, keyword and semantic retrieval, cross-session carry-over, persistent mood/relationship/style state, a validated rolling summary, and a session diary.
3. Voice and images are local and wired into the chat box. Kokoro-82M text-to-speech through a portable Node runtime, and Stable Diffusion through stable-diffusion.cpp on Vulkan — neither needs PyTorch, and both are reachable from a command or from the AI's own reply.
4. Plus engine work upstream hasn't released: llama.cpp b11130 with the new --load-mode flag surface, MTP/draft heads filtered out of the Model dropdown, and hardening so one broken extension can't take the app down.

2.Side-by-side comparison

Upstream column = what the public v4.9 release and its docs describe. Right column = this build.

Note on the "—" rows: those are features the core app genuinely doesn't provide, which is why they became extensions here. It isn't a knock — upstream deliberately keeps the core lean and points people at the community extensions directory.

3.The custom extension suite

Eight extensions, all in user_data/extensions/, all written for this build. Commands marked "no model in the loop" return their result directly, so they work reliably even with a small or thinking-disabled model.

4.Extension framework changes

Small code changes, big effect on how the app feels to use — and they're the reason the seven extensions read as one product instead of seven add-ons.

5.Engine & model-management work

llama.cpp b11377, with the old UI boxes still meaning the right thing

The bundled llama.cpp has been updated twice — to b11130 (0.4.1-dev) in September and to b11377 (0.5.0-dev) on 4 October, both Vulkan builds. The newer one brings vectorised Vulkan loads with an fp16 dot product, and probabilistic draft sampling for speculative decoding. That update removed the old --mmap / --mlock / --direct-io flags in favour of a single --load-mode. Rather than break the two checkboxes in the Model tab, the loader translates them:

neither box   -> no flag (auto)
no-mmap       -> --load-mode none
mlock         -> --load-mode mmap+mlock
both          -> --load-mode mlock

older binary  -> falls back to --no-mmap / --mlock (probed once per binary, cached)
extra flags   -> an explicit --load-mode wins; the translated one is suppressed

• The server's capabilities are probed once per binary path and cached, so an older build, a backup folder or a different fork keeps getting the flags it understands.
• The load log now reports the mode it chose, so a load line tells you which path was taken.
• The previous binary version is kept in a backup folder for instant rollback.

The llama.cpp build is visible in the Model tab

A small grey line under the model loader reads, for example, llama.cpp b11377 (0.5.0-dev). It is read from the binary itself once per session, so a screenshot is self-describing and a bug report needs no guessing about which build produced it.

Two ways a draft setting used to kill a load — both now caught

• A draft spec type with no draft model. The two settings live in different boxes and are applied separately, so one could be set while the other was empty — llama.cpp then read it as self-speculation against the main model and exited. The loader now says so plainly and starts without speculative decoding.
• A draft model that is the model itself. Easy to do by picking the wrong entry in the dropdown, and also possible when a second saved entry for the same model (written under a differently-cased name) quietly replaces the correct one. Treated as no draft, with the reason written into the log.
• The load line now reports spec_type and model_draft alongside gpu-layers, context and cache type, because the flags that cannot start a server were never the ones being printed.

A second model for general-purpose work

Alongside the Gemma 4 chat models there is now Qwen3.8-27B (GSQ-RCO, IQ3_S) with its vision projector and its own instruction template: 256k trained context, thinking by default, and around 36 tok/s at a q8_0 KV cache on this card. It is set up for document, code and analysis work rather than persona chat, which is why spec_type is none — its MTP layer is built into the model rather than supplied as a separate head.

Draft / MTP heads kept out of the Model dropdown

A speculative-decoding head (mtp-*.gguf, or a file whose GGUF metadata declares an *-assistant architecture) is not a chat model on its own, but it used to show up in the model list as if it were. Detection is metadata-first — the architecture is read straight out of the GGUF header, so renaming the file doesn't fool it — with a filename rule as a fallback.

• Applied to the Model dropdown only: the list is filtered, so you can't pick a head by mistake.
• The right-hand model-draft dropdown is deliberately left unfiltered, because that's where a head is actually used.
• The API's model list and the Training tab also keep the full list, and a head that's currently loaded is never hidden from its own dropdown.
• The same rule lives in the download-vetting tool, so a head is flagged before you decide to keep it.

Model-tab help text corrected

Some KV-cache types are ExLlamaV3-only and silently ignored on llama.cpp — the tooltip now says so instead of implying they apply to every backend.

6.Reliability fixes & reply hygiene

7.Performance engineering

The interesting one, and the thing most people building memory extensions get wrong.

• Summaries are generated in a background thread, so a long chat never stalls a live reply.
• Commands are answered without invoking the model at all (no tokens, no wait, works with thinking disabled).
• Embedding-based memory retrieval runs against a small local model served by llama.cpp — no Python ML stack loaded into the app process.

8.What came from upstream (not our work)

Worth stating on camera — it makes the comparison credible and gives credit where it's due.

• The application itself: the Gradio UI, chat/notebook/parameters/model/session tabs, Jinja2 prompt templates, character handling, streaming.
• All the loaders and backend switching, model metadata handling, and the portable packaging approach.
• The OpenAI/Anthropic-compatible API, tool calling (including MCP servers over HTTP and stdio), and the tool-parsing layer.
• Vision/multimodal input, text/PDF/docx attachments, gallery, character bias, autosave.
• LoRA training and evaluation, LaTeX rendering, syntax highlighting, themes, and the built-in docs.
• Web search plumbing, reasoning/thinking-channel handling, and the spec-decoding support the newer llama.cpp builds rely on.
• An image-generation tab (diffusers-based) — this build's image panel is a different engine wired into the same idea.

9.Portable setup, rollback & tooling

10.Recent work timeline

11.Demo order for the video

A sequence that shows the strongest things first and never leaves the chat box. If you'd rather not open a live chat at all, the sample output page already contains the finished version of every one of these moments.

1. Launch — double-click, window opens. Say the number: no install, no Python, no keys.
2. Memory basics — /remember a fact, then mention its keyword in normal conversation and show it surfacing. Follow with /memories.
3. Commands without a model — /file write notes.txt | … then /file read notes.txt. Point out these return instantly and can't be hallucinated.
4. Summarise — /sum20 in a long chat: clean [Summary] + five keywords, first person, no preamble.
5. Continuity state — /mood, /style, /relation: state that persists across sessions.
6. Diary — /diary then /diary read: a dated entry written from the session itself.
7. Verbatim recall — ask what was said earlier and show the log lines with real timestamps.
8. Voice — Speak last reply; change character; play again to show the voice follows. Mention the mic loop panel as next.
9. Images — /image a scene; then ask the AI for a picture and show the tag replaced by the render.
10. Extensions tab — show the tidy collapsed panels and that core panels can't be disabled by accident.
11. Model tab — show the draft head is absent from the model list but present in the draft list; point at the small llama.cpp b11377 line under the loader, and at the load line reporting its load mode, spec type and draft model.
12. Wrap — the honesty line: same app as upstream, newer engine, plus the memory/voice/image layer; credit upstream by name.

12.Credits & licence

• Upstream project: oobabooga/textgen — the application this build is based on. Stars, docs and the community extensions directory all live there.
• Licence: AGPL-3.0, inherited from upstream. Modifications are distributed under the same licence.
• Third-party components used by the extensions: llama.cpp (and its embedding server), stable-diffusion.cpp for image generation, Kokoro-82M via kokoro-js for speech, Whisper for speech-to-text, and bge-small-en-v1.5 for local embeddings. Each keeps its own licence; none of them phone home at runtime.
• No telemetry, no external requests in normal use — web search is opt-in per message and only fetches when asked.

Built as a personal project on top of upstream's work. Upstream feature claims here are based on the public v4.9 release notes and the project's own documentation; everything in the right-hand column is present in this install and was verified by running it.