Tag: memory

Memory in the Weights: Teaching a 4B Model New Facts While It Sleeps, Without Teaching It to Lie

A lab note on a wake–sleep memory for a small language model: a teacher distils the day's messages at night and the facts are trained into per-person experts behind a learned router that can stay shut. Dense memory hallucinated; routed memory kept the model's own abstention. With Qwen3-VL-4B the memory held 100 episodes at best; the first Spark-X2.6-4B seed still passes at 250, and the three-seed answer is not in yet. All of it runs on our Rust engine, on Metal and CUDA.