Tag: distillation

Memory in the Weights: Teaching a 4B Model New Facts While It Sleeps, Without Teaching It to Lie

A lab note on a wake–sleep memory for a small language model: a teacher distils the day's messages at night and the facts are trained into per-person experts behind a learned router that can stay shut. Dense memory hallucinated; routed memory kept the model's own abstention. With Qwen3-VL-4B the memory held 100 episodes at best; the first Spark-X2.6-4B seed still passes at 250, and the three-seed answer is not in yet. All of it runs on our Rust engine, on Metal and CUDA.

Sounds and Videos as Prompts: A 107M Adapter That Lets EmbeddingGemma 2 Drive Qwen-Image and Iris, Retrained on Permissive Data

We replaced the 8B text encoder of Qwen-Image 2.1 with the 0.74B EmbeddingGemma 2 plus a small adapter, so a sound or a short video can be referenced in a prompt like an image. Then we threw away every non-commercial training item and retrained on CC0 / CC BY data only. The permissive adapter is better on text, the best we have on video, and the licensing surprise was not in the data.