ThinkSound

ThinkSound composes the sound layer for silent video — footsteps, doors, engines, rain, room tone — the work a film crew records on a foley stage or assembles by hand. Its idea was to stop generating that layer in one blind step. A multimodal language model (a fine-tuned VideoLLaMA 2) first writes out reasoning about what is on screen, when it happens and how it should sound, and only then does an audio model follow that plan. The team introduced this to the task in June 2025, and the paper was accepted to the NeurIPS 2025 main conference. The system works in three stages. It first generates a coherent soundscape for the whole clip; then a viewer can click an object in the frame and have its sound refined separately; then the result can be edited with a written instruction. One model covers generation, refinement and editing. The published measurements make the case and also show its limit. Against MMAudio, the strongest comparable system, ThinkSound reports a better distribution distance (34.56 against 43.26) and higher listener scores for both audio quality and alignment (4.02 and 4.18 against 3.84 and 3.97; the original recordings score 4.37 and 4.56). It is also close to three times faster per clip — 1.07 seconds against 3.01 — despite being the larger model. Where it does not win is synchronisation: its timing error, 0.46, is marginally worse than MMAudio's 0.44, so the reasoning buys understanding of the scene rather than precision. Turning the reasoning off costs 13 percent of the quality score and saves only 0.09 seconds. The generator is a 1.3-billion-parameter multi-stream diffusion transformer, 21 layers deep — 14 of them keeping separate weights per modality, 7 shared. The paper also documents medium (724M) and small (533M) configurations that were not released. What is published is the large one: a 21.06 GB checkpoint, a 5.73 GB lightweight version added a week later, plus a 2.52 GB audio autoencoder and a 950 MB synchronisation module. The licence needs reading twice, because the label and the text disagree. Hugging Face marks the repository Apache 2.0 and it does ship the full Apache text — but the notice at the foot of that same file, and the project README, both state that code, models and dataset are for research and education only and that commercial use is not permitted. The audio autoencoder is a fine-tune of Stability AI's Stable Audio Open and stays under the Stability AI Community License, which forbids redistribution under any other licence. Since nothing decodes to audio without that component, the restriction covers the working system, not a detachable part of it. Hugging Face records no downloads for the repository, but that number should be ignored here: the counter only registers standard model files, and this release ships PyTorch checkpoints. MMAudio's repository reports zero the same way. The GitHub project has over 1,300 stars and third-party integrations, including a ComfyUI wrapper.

▸News

The label says Apache. One file out of four says no.

9/5/2026 · Hugging Face LICENSE.md, README repozytorium ThinkSound (GitHub), licencja Stability AI

▸Videos

No videos yet.

▸Reviews