Sounds and Videos as Prompts: A 107M Adapter That Lets EmbeddingGemma 2 Drive Qwen-Image and Iris, Retrained on Permissive Data
Qwen-Image takes reference images in its prompts: image1, image2, up to ten. We wanted the same sentence to accept audio1 and video1. Not through a separate audio model bolted on the side, but through the one encoder that already embeds text, images, sounds and video frames in a single language model: Google's EmbeddingGemma 2 (EG2, 0.74B parameters).
Qwen-Image 2.1 conditions its 7B diffusion transformer on the hidden states of an 8B Qwen3-VL text encoder. If a small adapter can turn EG2's per-token states into the sequence the DiT expects, then whatever EG2 can embed becomes a prompt token. The adapter is 107M parameters, eight transformer layers, 768 in, 4096 out. It took three training rounds, one Rust/Metal engine, and a licensing audit on release day to get here.
This article covers the result, the permissive retrain, the numbers against the earlier adapters, and what we would do differently.
What it does
Everything below is the engine running on an M5 Max, no Python in the loop. A prompt like a photo of audio1 with a four-second recording of an acoustic guitar as audio1:
These are held-out FSD50K evaluation clips the adapter never trained on, each the sole reference, 1024², Qwen-Image Turbo, seed 7. No text other than a photo of audio1. The sound enters the DiT as 25 soft tokens per second through EG2's audio tower, then through the adapter, exactly like text would. There is no spectrogram image and no VAE latent for the audio: that is the point of the proof.
Video works the same way, with video1 filled by up to six frames of a clip:
Those four are MSR-VTT clips, 512² at 8 steps on the Mac. The captions under them are the dataset's own descriptions of the source clips, not prompts we wrote.
And it fails in an informative way:
The adapter gets the kind of sound right (bowed instrument, brass, something mechanical on a road) and misses the exact object. Abstract and texture sounds (wind, a zipper, a typewriter, a running tap) score zero. Sounds with an obvious visual score at or near 100 %.
Why a third training round
The first adapter (v1) worked. Its training data did not survive a release review:
| source | used for | problem |
|---|---|---|
| ShareGPT4V captions (1.25M) | stage 1 text distillation | CC BY-NC 4.0 |
| ESC-50 sound clips | reference alignment | CC BY-NC 3.0 |
| MSR-VTT video clips | reference alignment | research-only terms |
| Pexels videos | considered | license forbids training ML models |
| YouTube | considered | no |
A model you cannot publish is a demo. So v3 replaced every item:
| modality | source | license | count |
|---|---|---|---|
| text | COCO captions, Localized Narratives, DOCCI (x8), DiffusionDB prompts | CC BY 4.0 / CC0 | 1.95M rows |
| audio | FSD50K dev clips, per-clip license from the Zenodo metadata, CC0 and CC BY 3.0 only (by-nc and sampling+ excluded) | CC0 / CC BY 3.0 | 14,977 clips, 19.6 h, all 200 classes |
| video | Wikimedia Commons WebM from the CC-Zero / CC-BY-3.0 / CC-BY-4.0 categories, six frames each, captioned by Qwen3-VL-8B-Instruct (Apache-2.0) | CC0 / CC BY | 4,473 clips after filters |
The attribution list is generated from the manifests, never written by hand: 19,582 lines, one per Freesound clip and Commons file, plus the dataset-level credits. It ships with the weights. Commons license tags are uploader-asserted, and the model card says so.
Training stayed the same shape as v1: stage 1 distills EG2 token states onto the teacher encoder's last-layer hidden states in per-channel standardized units (teacher channel std spans 6 to 134, with two massive-activation channels that would otherwise eat the whole loss); stage 2 distills velocities through the frozen DiT over fourteen resolution buckets from 512² to 2048x1152; phase D aligns references, where the student sees EG2 text with an <|audio|> or <|video|> slot filled by a clip and the teacher sees the caption that names it (tail MSE, pooled MSE, InfoNCE). One change mattered: v3 keeps the stage-1 text loss in the phase-D mix. More on why below.
Stage 1 ran 20k steps on one RTX 3090 (cosine 0.902 at the end). Stage 2 + phase D ran 6,500 steps on the same card, in five-minute chunks, because the GPU is shared.
Results against the earlier adapters
Text prompts, adapter versus the Qwen3-VL teacher on the same 62 prompts and seeds, 40 steps, paired per prompt with bootstrap intervals. Closer to zero is better; PickScore is the human-preference proxy, GenEval the compositional check (object presence, count, colour, position, through OWLv2):
| adapter | training data | SigLIP2 gap | CLIP gap | PickScore gap | GenEval pass | OCR word acc. |
|---|---|---|---|---|---|---|
| v1 final (step 3500) | mixed, NC | -2.42 | -2.43 | -0.83 | 55.6 % | 16.7 % |
| v1 step 1500 (the old shipping pick) | mixed, NC | -1.71 | -1.71 | -0.85 | 69.4 % | 16.7 % |
| v3 final | CC0 / CC BY only | -1.45 [-2.36, -0.68] | -1.51 | -0.52 | 69.4 % | 41.7 % |
| teacher | 0 | 0 | 0 | 77.8 % | 83.3 % |
The permissive adapter closes about 40 % of v1's gap to the teacher and is the first one where rendered text is usable at all. GenEval is no longer significantly different from the teacher. Spatial relations ("a cat to the left of a dog") remain the one clear loss, 50 % against 100 % on six prompts, the same weakness every adapter has shown.
References, zero-shot retrieval over held-out clips:
| test | v1 best | v3 | chance |
|---|---|---|---|
MSR-VTT video1, 100 clips, SigLIP2 R@1 / R@5 / R@10 |
41 / 77 / 83 % | 67 / 95 / 98 % | 1 / 5 / 10 % |
| the adapter's own text ceiling on those captions | 50 / 78 / 83 % | ||
ESC-50 audio1, 100 clips, 50-way CLIP top-1 / top-5 |
47 / 72 % | 24 / 48 % | 2 / 10 % |
FSD50K eval audio1, 100 clips, 100-way top-1 / top-5 / median rank |
not run | 14 / 28 % / 19.5 | 1 / 5 % / 50.5 |
Video following beats the adapter's own text ceiling: an image generated from the clip retrieves the clip's caption more reliably than an image generated from the caption does. The reference carries information the caption does not.
The ESC-50 row looks like a regression and is not a fair comparison. v1 trained on ESC-50 clips of exactly those 50 classes. v3 never saw ESC-50, because it is CC BY-NC. Its 24 % is zero-shot transfer against 2 % chance. The like-for-like audio number is FSD50K eval, which neither adapter trained on and only v3 was measured on: 14 % top-1 over 100 classes with a median rank under 20, under an adapter text ceiling of 81 %. Sounds reach roughly a sixth of what text reaches at top-1, and the per-class table says exactly where the gap is: instruments, birds, clapping, fireworks, laughter, singing and coughing at 100 %; wind, zipper, typewriter, tap, writing at 0 %.
Why one adapter instead of two
v1 shipped as a pair: a "balanced" checkpoint and a "max-sound" one, because every phase-D step improved references and cost text (GenEval 69 % at step 1500, 56 % at step 3500). v3 kept the stage-1 text objective in the alignment mix and the text numbers stayed flat for 6,500 steps. We benchmarked four v3 checkpoints to be sure:
| checkpoint | SigLIP2 gap | GenEval | OCR | references |
|---|---|---|---|---|
| stage 1 only (no alignment) | -1.26 | 75.0 % | 50 % | none |
| phase D step 2000 | -1.50 | 58.3 % | 8 % | not tested |
| phase D step 4000 | -2.25 | 66.7 % | 67 % | not tested |
| phase D step 6500, final | -1.45 | 69.4 % | 42 % | video R@1 67 %, audio 24 % |
The final checkpoint is inside every confidence interval of the text-only one. A second "text-only" adapter would buy noise.
The same recipe on Iris-3B
Iris-3B is a 3B pixel-space flow model (Apache-2.0) conditioned by a Qwen3-VL-4B encoder. The same EG2 adapter idea, with a 2560-d output and five tail rows, trained on Commons photos, Iris's own Turbo outputs and 1,173 OCR-verified synthetic text renders, goes against Iris's native conditioning at 1024², 30 prompts x 3 seeds:
| native Qwen3-VL | EG2 adapter (permissive) | |
|---|---|---|
| SigLIP2 paired gap | 0 | -0.29 [-1.09, 0.41] |
| GenEval pass (24 prompts) | 86.1 % | 75.0 % |
| OCR word accuracy (6 prompts) | 100 % | 22.2 % |
Statistically indistinguishable on alignment, behind on rendered text. The text-render share in the training mix is what took OCR from 2.8 % to 22 %; the row below is the kind of prompt that still separates them (native first, then the four adapter variants):
In the Rust/Metal engine that adapter costs nothing you can see: Iris Turbo 1024² at 2.2 to 2.4 s per image warm, 7.3 GB on the device, EG2 text stage 10 ms warm. It ships inside OakOliver Studio today as the iris-turbo-metal-eg2 and iris-base-metal-eg2 rows, as a bundle with a manifest and sha256s, installed through the same pin check as every other engine.
Engine numbers
The Qwen-Image side runs in qwen-image-rs, our Rust engine with hand-written Metal kernels, bf16 on the device and f32 where it matters. Swapping the 8B encoder for EG2 plus the adapter:
| stage | Qwen3-VL encoder | EG2 + adapter |
|---|---|---|
| text stage, warm, 16 / 64 / 256 tokens | 81 / 88 / 148 ms | 8.1 / 11.7 / 33.1 ms |
| peak memory, 1024² 8 steps | 32.6 GB | 21.7 GB |
| end-to-end, fresh process / warm request | 14.57 / 14.63 s | 14.99 / 14.39 s |
The DiT dominates, so end-to-end time is a wash; memory and the text stage are not. The new bottleneck for reference prompts is the host: EG2's audio tower runs in f32 on the CPU at about 2 s per two-second clip, and the vision tower about 9.6 s per four-frame video. Moving those to Metal is the next engine task.
Up to nine audioK and nine videoK references per prompt, path#t=start,end for a time range, #ch=split for stereo, mp3 / flac / ogg / m4a and video soundtracks through ffmpeg, and batch / serve modes that keep the weights resident.
The licensing surprise was not in the data
On release day we checked the licence of every frozen model through the Hub API, which we should have done on day one:
| model | role | licence |
|---|---|---|
| google/embeddinggemma-2 | encoder | Apache-2.0 |
| Qwen/Qwen3-VL-8B-Instruct | video captioner | Apache-2.0 |
| speridlabs/iris-3b | Iris target | Apache-2.0 |
| Qwen/Qwen-Image-2.1 | Qwen-Image target and teacher | Qwen Research License, non-commercial |
Two things we had assumed were wrong. EG2 is not under the Gemma Terms of Use; it is plain Apache-2.0, which removes the notice-and-restrictions chain we had planned for. And Qwen-Image 2.1 dropped the Apache-2.0 licence its predecessor had. Its text encoder is not the Apache Qwen3-VL-8B release either: the safetensors shards have different hashes, so the teacher we distilled from is itself research-licensed.
So the permissive retrain changed what the data allows, and the target model decides the rest. The Qwen-Image adapter is a derivative of research-only Materials and is released for research and evaluation, with Alibaba's commercial licence required beyond that. The Iris adapter has no such limit: Apache-2.0 model, Apache-2.0 encoder, CC0 / CC BY data, so it goes out under Apache-2.0 with the attribution file.
The lesson is blunt: the frozen target decides the licence of the adapter, not the training set. Check it before the first training step.
What we would do differently
Keep the text objective in every phase. The single change that turned a two-adapter release into one.
Compare like for like, and say so. The ESC-50 number would have read as a regression in a summary table. The per-class FSD50K table is what tells us which data to add next.
Validate two to four images at final settings before any batch. Our first 512² 8-step benchmark produced images the reviewer called terrible. Sixty images at the wrong settings cost an hour of GPU and a day of trust.
PSNR only for engine parity. It is valid between images of the same content (Huynh-Thu and Ghanbari, 2008), so it gates Metal against the f32 reference and nothing else. Adapter quality is judged on SigLIP2, CLIP, PickScore, GenEval-style checks and OCR, paired against the teacher with bootstrap intervals.
Short lock tickets, resumable chunks. The training GPU is shared with another project through a lock queue, and the rule we agreed on is exclusive tickets of at most five minutes. Every benchmark became a chain of five-prompt chunks that resume from a manifest, with a HOLD file to pause between chunks. Two shell bugs cost an afternoon: a set -- that clobbered the function's own arguments, and a timeout prefix that the lock wrapper turned into python timeout because it prepends the interpreter. A ten-line Python wrapper that enforces a wall-clock budget replaced it.
Prune checkpoints the day a phase finishes. The rig fell to 37 GB free and the Mac data volume to 15 GB in the same week.
What is next
The Iris adapter is on its way to Hugging Face under Apache-2.0. The Qwen-Image adapter follows as a research release, with the model card above. The engine work queue: audio and vision towers on Metal, an external anchor by running the public ImageBind + Stable Diffusion recipe through the same ESC-50 / MSR-VTT bench, and more texture-like sounds in the training mix, since that is where the per-class table says the gap is.
The adapters, the data pipeline and every benchmark report live in the eg2-qwen-image-rs repository; the engines are qwen-image-rs and iris-rs. Numbers in this article are from the reports committed there on 2026-10-21.