Video-RobinExploring Textual Guidance for
Video-to-Music Generation Models
Supplementary material · audio samples and instruction-following examples
Supplementary material · audio samples and instruction-following examples
“A funky, energetic 98 BPM electro-hip-hop groove in the key of A minor featuring aggressive synth basslines, punchy 808 kick drums, crisp snares, and metallic off-beat claps. Layered with bright, chopped brass synths, glitchy hi-hat rolls, and an upbeat street-dance rhythm.”
A diffusion-autoregressive
text + video-to-music model
Mel-spectrogram of the generated audio · dashed lines mark detected beats
Described back by two models shown only the result
“The generated music features an intense, fast-paced track… composed of prominently layered body percussion,… closely mirror the performer’s physical movements… atmosphere that perfectly amplifies the powerful, martial arts-inspired energy of the video.”
“This instrumental piece delivers a high-energy funk-infused electronic hip-hop vibe. The production centers on a deep synth bass… tight, side-chain-compressed 808 kicks… overall mood is confident and danceable, evoking urban nightlife scenes where grooves dominate the floor.”
Current video-conditional music generators expose a narrow control interface, leaving the creator unable to specify the music a scored video should carry. The same clip is consistent with many combinations of genre, instrumentation, key and tempo, and only the creator’s intent selects among them. This also goes unmeasured, since the field reports audio quality and audio–visual similarity but not whether a stated attribute was realised. We explore this gap with ReelBench, and develop Video-Robin by attaching a video adapter to a patch-based diffusion autoregressive music generator. It obtains the lowest FAD in our comparison at the shortest inference time, on 1.0 B parameters and 311 hours of paired data, while following tempo close to the strongest baseline.
Paired clips are filtered for musical usability and then re-captioned as prompts, so that every training pair states the attributes a creator would want to control.
Video-Robin against video-only and text-conditioned baselines on the same clips.
A meditative ambient-droning track at an indeterminate, very slow pace with a calm and contemplative mood, featuring layered evolving synth pads in a warm, organic timbre, a deep sustained sub-bass drone, crystalline bells, airy whistles, and faint granular noise, all arranged in a wide stereo field with generous reverb and delay to evoke vastness and timelessness, while subtly underlining a slightly melancholic undertone that complements the introspective and expansive atmosphere.
Every system received the prompt above alongside the video. Scroll sideways for the remaining systems.
A mid-tempo electronic track in d minor featuring a spacious, reverb-drenched synth pad and subtle field recordings, establishing a contemplative, slightly melancholic atmosphere, with a steady, slow rhythm that maintains a consistent, relaxed pace, layered with a muted, low-pass-filtered kick and soft, syncopated hi-hat pattern, and a warm, rounded bass synth that anchors the groove, evolving into arpeggiated synth lines and delicate percussive clicks that create gentle forward motion, all while evoking serenity and tranquility through a culturally neutral soundscape that invites quiet contemplation and a deep sense of natural stillness.
Every system received the prompt above alongside the video. Scroll sideways for the remaining systems.
A contemplative ambient track at a consistent medium tempo, featuring C Major chords and a sparse, reverberant piano motif layered over evolving low-pass filtered synth pads, creating a spacious, introspective atmosphere that subtly contrasts with fluctuating emotional expressions in the video, with delicate arpeggiated synths and faint high-frequency textures adding gentle motion while maintaining a calm, meditative mood, supported by a barely perceptible filtered percussive pulse for rhythmic anchor, and concluding with a lingering resonant chord in a clean, airy mix that emphasizes quiet reflection.
Every system received the prompt above alongside the video. Audio is cropped to an identical length and muxed onto the identical video crop. Scroll sideways for the remaining systems.
An ambient-electronic track at 130.43 BPM in A Minor, featuring lush evolving synth pads, a deep sustained synth bass, and a bright arpeggiated synth lead that carries a hopeful, yearning melody, set against a crisp electronic drum kit with a steady four-on-the-floor kick, tight snare/clap accents, and subtle hi-hat shuffles, all layered with reverb and delay to create a dreamy, introspective atmosphere, blending downtempo chillout textures with subtle melodic pop sensibility, conveying a sense of gentle optimism and introspection, while maintaining a calm and steady rhythm that subtly shifts in tonal center through the Am7-to-A progression, evoking a serene, uplifting mood rooted in modern ambient and chillout traditions.
Every system received the prompt above alongside the video. Audio is cropped to an identical length and muxed onto the identical video crop. Scroll sideways for the remaining systems.
A slow ambient track at 75 BPM in F Major, featuring spacious evolving synth pads, subtle field recordings, delicate low-frequency pulses, faint high-frequency arpeggios, and a soft filtered percussive loop, creating a meditative and introspective atmosphere that evokes calm, wonder, and subtle introspection, with a universally accessible, ethereal quality that shifts between peacefulness and contemplative depth.
Every system received the prompt above alongside the video. Audio is cropped to an identical length and muxed onto the identical video crop. Scroll sideways for the remaining systems.
A slow ambient track at 60 BPM in F minor, featuring evolving synth pads, subtle field recordings, deep resonant sub-bass, delicate high-frequency arpeggios, and soft filtered percussive clicks, evoking a calm, introspective atmosphere rooted in Hindu mythology and infused with a meditative, contemplative mood that gradually builds into a subtle, uplifting climax, reflecting the celebratory and spiritually resonant theme of Radha and Krishna while maintaining a relaxed, ethereal production style.
Every system received the prompt above alongside the video. Audio is cropped to an identical length and muxed onto the identical video crop. Scroll sideways for the remaining systems.
A serene ambient track at a slow, steady 70 BPM in D minor, featuring downtempo electronic textures, subtle chillwave sensibility, evolving synth pads, delicate field recordings, a soft low-frequency pulse, shimmering arpeggiated synths, distant reverberant percussive clicks, and faint processed vocal-like textures, all layered to create a calm, introspective soundscape that evokes tranquility, steadfast love, and a meditative, emotionally layered atmosphere with a South Asian cultural suggestion through the text overlay.
Every system received the prompt above alongside the video. Audio is cropped to an identical length and muxed onto the identical video crop. Scroll sideways for the remaining systems.
An epic cinematic orchestral track at ≈143 BPM in C minor, featuring lush strings, powerful brass, resonant timpani, and a wordless choir with a wide, deep stereo field, delivering a dark yet uplifting soundscape that blends traditional symphonic writing with modern hybrid elements, driven by a consistent, driving 4/4 rhythm with moments of tension and release through harmonic shifts between C minor, G minor, F minor, and a bright C major lift, evoking a heroic, adventurous journey filled with determination and optimism, while maintaining a grand, searching mood that aligns with the theme of overcoming challenges and achieving personal triumph.
Every system received the prompt above alongside the video. Audio is cropped to an identical length and muxed onto the identical video crop. Scroll sideways for the remaining systems.
ReelBench, with systems grouped by capability class. Bold marks the best and underline the second best across all systems. Ground truth is the reference recording, not a competitor, so it is excluded from those rankings.
| Model | Params | Infer. (s) ↓ | FAD ↓ | FD ↓ | KL ↓ | IS ↑ | IB ↑ | Dens. ↑ | Cov. ↑ |
|---|---|---|---|---|---|---|---|---|---|
| Ground truth | — | — | — | — | — | — | 0.1417 | 0.9900 | 0.8800 |
| Unified / omni-model LLMs — understanding and generation | |||||||||
| MuMu-LLaMA | 9.8 B | 22.87 | 4.5767 | 27.4208 | 1.5301 | 1.6499 ± 0.0567 | 0.0722 | 0.1094 | 0.2761 |
| Audio-Omni | 8.1 B | 20.67 | 2.3986 | 18.5889 | 1.3308 | 1.5831 ± 0.1275 | 0.1499 | 0.3600 | 0.0850 |
| Any-to-audio generation: {text, video, image, audio} → {general audio, music} | |||||||||
| AudioX | 1.6 B | 12.49 | 4.5702 | 26.0165 | 1.4381 | 1.6950 ± 0.1631 | 0.1490 | 0.1380 | 0.0379 |
| Video-to-music generation | |||||||||
| GVMGen | 2.3 B | 28.25 | 3.5729 | 16.2638 | 1.5573 | 1.7085 ± 0.0281 | 0.0957 | 0.0835 | 0.3881 |
| VidMuse | 2.0 B | 41.55 | 2.3022 | 14.5385 | 1.3194 | 1.4549 ± 0.0281 | 0.1233 | 0.1377 | 0.5213 |
| CMT† | 49 M | 12.70 | 8.7522 | 37.7945 | 1.7329 | 1.2243 ± 0.0147 | 0.1119 | 0.1084 | 0.0614 |
| Video2Music† | 34 M | 8.55 | 22.6459 | 73.0670 | 1.8839 | 1.0233 ± 0.0014 | 0.0473 | 0.1647 | 0.0084 |
| Text + video-to-music generation (ours) | |||||||||
| Video-Robin ours | 1.0 B | 3.87 | 1.5110 | 10.9020 | 1.2556 | 2.0586 ± 0.0472 | 0.1017 | 0.1647 | 0.5259 |
† Symbolic (MIDI) systems, whose audio metrics are computed on a fixed-synthesiser rendering. Params and inference time describe the system; the remaining columns score its output. IS is reported with its standard deviation.
Pairwise A/B listening comparisons between six systems. Each cell is the share of comparisons in which the row system was preferred over the column system; every opposing pair sums to 100. The bar on the right of each row is that system’s mean win rate across its five opponents, so a row can be read straight across from its matchups to its overall standing.
Opponent →
Each prompt states a tempo and a key. We recover both from the generated audio and compare them against what was asked. Adherence to key is markedly less reliable than adherence to tempo. All four clips are Video-Robin outputs; scroll sideways if they overflow.
Requested tempo and key are both realised.
120 BPM→128 BPMC Major→C majorThe tempo lands, the key does not — the most common failure mode across every system we evaluate.
143 BPM→144 BPMB♭ minor→not detectedThe key lands, but the music runs at a different tempo than the one asked for.
70 BPM→115 BPMA Major→A majorNeither the requested tempo nor the requested key is realised.
150 BPM→62 BPMA minor→G majorAll systems per clip, on a shared decibel scale. Vertical striations are transients — percussive onsets and note attacks; systems that smear them horizontally produce sustained texture rather than rhythmic content.
Further Video-Robin outputs, scored onto their source clips.