Add sound to a scene
Choreograph sound and motion, then export the same scene with audio
This guide shows how to add a local sound to a scene and export that same scene with audio. It assumes you can already make a scene and use the Player. A scene is an Effect: yielding an audio resource declares its dependency, and yielding transport or animation effects changes its frame data.
Declare and play a track
Create a resource with a literal id, then yield it inside your scene. play
starts a new Audio instance and returns immediately. It does not keep the
scene alive, so wait explicitly for the part of the sound you want to show.
import * as Audio from "effect-motion/Audio";
import * as Scene from "effect-motion/Scene";
export const Theme = Audio.Audio("theme");
export const scene = Scene.make(function* () {
const theme = yield* Theme;
const sourceDuration = yield* Audio.duration(Theme);
const playing = yield* Audio.play(theme, { gain: 0 });
yield* Audio.fadeIn(playing, "400 millis");
yield* Scene.sleep(sourceDuration);
yield* Audio.stop(playing);
});duration is the whole source length, even when playback starts at a
nonzero from position. For the remaining time of a nonlooping play, subtract
that starting offset from the source duration. The query requires prepared
metadata; declaring or playing a track alone does not read its bytes during
frame generation.
Prepare the bytes for each runtime
The Player needs a layer built with prepareAudio from
@effect-motion/react. It decodes the bytes before the scene runs and
provides both playback data and duration metadata. Use a user gesture to
start audible playback. If the browser blocks autoplay, the Player holds at
the first audible frame until Play is pressed.
import { Player, prepareAudio } from "@effect-motion/react";
import * as Layer from "effect/Layer";
import * as Resource from "effect-motion/Resource";
import { Drift, Groove, Hit, Riser, scene, Tick } from "./audio.scene";
const audio = Layer.mergeAll(
prepareAudio(Groove, Resource.fetchBytes("/audio-groove.wav")),
prepareAudio(Drift, Resource.fetchBytes("/audio-drift.wav")),
prepareAudio(Riser, Resource.fetchBytes("/audio-riser.wav")),
prepareAudio(Hit, Resource.fetchBytes("/audio-hit.wav")),
prepareAudio(Tick, Resource.fetchBytes("/audio-tick.wav")),
);
export function Preview() {
return <Player scene={scene} renderLayers={audio} />;
}In Node, use Video.prepareAudio to inspect those same bytes before frame
generation and provide the resulting layer to Video.render. The
audio.render.ts next to the example below is a complete CLI entrypoint:
bun packages/cli/src/bin.ts render apps/docs/examples/audio.render.tsIt writes tmp/audio-preview.mp4 from the exact scene the Player displays.
Install the export and CLI packages in your own project to use this pattern
outside the repository. If the audio pass fails, export raises EncodeError
and can leave a video-only file at the output path; that file is an incomplete
render.
Control the sound
gain is a linear multiplier: 0 is silent and 1 is the source level. Animate
it with Motion.tween/tweenTo on { gain }, or use Audio.fade/fadeTo.
fadeIn starts at 0 and reaches 1; fadeOut reaches 0 while the cursor keeps
running, so pause or stop afterwards if needed. crossfade fades one Audio
instance out while another fades in. pause, resume, seek, and stop
control the returned instance; stop also rewinds it. Set loop: true in
play to wrap at the source end.
For a timed clip, pass play(theme, { duration: "1 second" }). It pauses at
that scene-time deadline, even if its calling branch has finished. The clip
holds the scene open like a fork started in that branch; a shorter-lived
parent branch can end that hold. A later pause, resume, seek, stop, or direct
transport write cancels the earlier clip deadline. Consult the
Audio API reference for the
full signatures.
The example below is a 15.5-second music ident at 120 BPM, built as a
sequence of shots. A four-tick count-in collapses into a dot; the groove
drops and floods the frame with type, then with pulsing columns. A riser
folds the columns into a blade while Audio.crossfade moves from the groove
to a drifting pad over exactly the riser's length. The camera then flies
through a tunnel of rings, everything pulls to a point on a held breath, and
the final hit reveals a lockup that fades out with the hit's chord.
The durations do the timing. The groove loop is exactly one bar, so
Audio.duration(Groove) sets the beat every cue waits on; the riser's
duration sets the transition and the hit's duration sets the ending. The
five small PCM WAVs are synthesized from oscillators and seeded noise by
audio.synth.ts next to the example; they contain no sampled or downloaded
material. Their levels are mixed in that script, so the scene changes gain
only for the accented last tick and the crossfade.
The picture changes are scheduled in scene time alongside the sound cues. They do not inspect the waveform or react to microphone input. WAV works in the verified browser and Node export paths here. Browser decoding depends on the browser; test other formats in both the browser and ffmpeg before relying on them. Audio analysis and editable DAW export are future work.
import { Duration, Effect, Schedule } from "effect";import * as Audio from "effect-motion/Audio";import * as Color from "effect-motion/Color";import * as Entity from "effect-motion/Entity";import type * as Instance from "effect-motion/Instance";import * as Motion from "effect-motion/Motion";import * as Scene from "effect-motion/Scene";// "One beat": a 15.5-second music ident at 120 BPM. A count-in becomes a hot// drop, the drop folds into a sweep, the sweep crossfades into a deep drift,// and everything collapses into a lockup on the final hit.//// Every cue is scheduled in scene time. The groove loop is exactly one bar,// so its prepared duration sets the grid; nothing reads the waveform.const ink = Color.hex("#0e0d12");const coral = Color.hex("#ff4b36");const cream = Color.hex("#f6ecd9");const night = Color.hex("#0a1433");const cyan = Color.hex("#43e4ff");const ice = Color.hex("#d9f7ff");const blue = Color.hex("#2448c8");const none = Color.rgba(0, 0, 0, 0);// Original synthesized WAVs (see audio.synth.ts). Browser and Node provide// the same five tracks to this one scene.export const Groove = Audio.Audio("groove");export const Drift = Audio.Audio("drift");export const Riser = Audio.Audio("riser");export const Hit = Audio.Audio("hit");export const Tick = Audio.Audio("tick");const WIDTH = 1280;const HEIGHT = 720;const hide = <Tag extends "Group" | "Hud" | "Circle" | "Rect" | "Text">( instance: Instance.Instance<Tag>,) => Scene.update(instance, (data) => ({ ...data, visible: false }));const show = <Tag extends "Group" | "Hud" | "Circle" | "Rect" | "Text">( instance: Instance.Instance<Tag>,) => Scene.update(instance, (data) => ({ ...data, visible: true }));const word = (text: string, fontSize: number, fillColor: Color.Color) => Scene.instantiate("Text", { text, fontSize, fillColor, textAnchor: "middle", baseline: "middle", });export const scene = Scene.make( "audio", function* () { const tracks = { groove: yield* Groove, drift: yield* Drift, riser: yield* Riser, hit: yield* Hit, tick: yield* Tick, }; // The groove is one bar long: its duration is the scene's clock. const bar = Duration.toMillis(yield* Audio.duration(Groove)); const beat = bar / 4; const sweep = yield* Audio.duration(Riser); const ring = yield* Audio.duration(Hit); // Start the moves together and hold exactly one beat. const onBeat = <R>( ...moves: ReadonlyArray<Effect.Effect<unknown, never, R>> ) => Scene.all([...moves, Scene.sleep(beat)]); // ── Cast. Later shots are built up front and hidden until their cue. const count = yield* Scene.instantiate("Group", {}); const numeral = yield* word("1", 460, cream); const marks: Instance.Instance<"Rect">[] = []; for (let i = 0; i < 4; i++) { marks.push( yield* Scene.instantiate("Rect", { position: Entity.vec3({ x: (i - 1.5) * 64, y: -300 }), width: 40, height: 6, fillColor: Color.hex("#34303b"), }), ); } for (const child of [numeral, ...marks]) { yield* Scene.appendChild(count, child); } const drop = yield* Scene.instantiate("Group", {}); const field = yield* Scene.instantiate("Circle", { radius: 0, fillColor: coral, }); const streaks = [ yield* Scene.instantiate("Rect", { position: Entity.vec3({ x: -1500, y: 250 }), width: 900, height: 18, fillColor: ink, }), yield* Scene.instantiate("Rect", { position: Entity.vec3({ x: 1500, y: -250 }), width: 900, height: 18, fillColor: ink, }), ]; const shout = yield* word(" ", 230, ink); for (const child of [field, ...streaks, shout]) { yield* Scene.appendChild(drop, child); } const pulse = yield* Scene.instantiate("Group", { visible: false }); const columns: Instance.Instance<"Rect">[] = []; for (let i = 0; i < 8; i++) { const column = yield* Scene.instantiate("Rect", { position: Entity.vec3({ x: (i - 3.5) * (WIDTH / 8) }), scale: Entity.vec3({ x: 1, y: 0, z: 1 }), width: WIDTH / 8, height: HEIGHT, fillColor: i % 2 === 0 ? cream : coral, }); yield* Scene.appendChild(pulse, column); columns.push(column); } const pulseWord = yield* word("PULSE", 300, ink); yield* hide(pulseWord); // center-out pairs, so each beat ripples from the middle const pairs = [0, 1, 2, 3].map((d) => [ columns[3 - d] ?? columns[0], columns[4 + d] ?? columns[7], ]); const backdrop = yield* Scene.instantiate("Rect", { position: Entity.vec3({ z: -6000 }), scale: Entity.vec3({ x: 1, y: 0, z: 1 }), width: WIDTH * 5, height: HEIGHT * 5, fillColor: night, }); const tunnel = yield* Scene.instantiate("Group", { opacity: 0 }); const rings: Instance.Instance<"Circle">[] = []; for (let i = 0; i < 11; i++) { const r = yield* Scene.instantiate("Circle", { position: Entity.vec3({ z: -400 * i }), radius: 330, fillColor: none, strokeColor: i % 3 === 0 ? cyan : blue, strokeWidth: 5, }); yield* Scene.appendChild(tunnel, r); rings.push(r); } const diamond = yield* Scene.instantiate("Rect", { position: Entity.vec3({ z: -4400 }), rotation: Entity.vec3({ z: Math.PI / 4 }), width: 180, height: 180, fillColor: cyan, }); yield* Scene.appendChild(tunnel, diamond); // Flat type over the moving camera. const title = yield* Scene.instantiate("Hud", { visible: false }); const letters: Instance.Instance<"Text">[] = []; for (const [i, letter] of [..."DRIFT"].entries()) { const t = yield* Scene.instantiate("Text", { text: letter, position: Entity.vec3({ x: (i - 2) * 170, y: -265 }), fontSize: 180, fillColor: ice, textAnchor: "middle", baseline: "middle", opacity: 0, }); yield* Scene.appendChild(title, t); letters.push(t); } const caption = yield* Scene.instantiate("Text", { text: "1 2 0 B P M · A M I N O R", position: Entity.vec3({ y: 285 }), fontSize: 24, fillColor: cyan, textAnchor: "middle", baseline: "middle", opacity: 0, }); yield* Scene.appendChild(title, caption); const lockup = yield* Scene.instantiate("Hud", { visible: false }); const flood = yield* Scene.instantiate("Circle", { radius: 0, fillColor: cream, }); const sun = yield* Scene.instantiate("Circle", { position: Entity.vec3({ x: -430, y: 14 }), scale: Entity.vec3({ x: 0, y: 0, z: 1 }), radius: 54, fillColor: coral, }); const gem = yield* Scene.instantiate("Rect", { position: Entity.vec3({ x: -305, y: 14 }), scale: Entity.vec3({ x: 0, y: 0, z: 1 }), width: 78, height: 78, fillColor: blue, }); const wordmark = yield* Scene.instantiate("Text", { text: "effect-motion", position: Entity.vec3({ x: -180, y: 24 }), fontSize: 108, fillColor: ink, textAnchor: "start", baseline: "middle", opacity: 0, }); const tagline = yield* Scene.instantiate("Text", { text: "sound, in scene time", position: Entity.vec3({ x: -176, y: -62 }), fontSize: 40, fillColor: Color.hex("#5c5249"), textAnchor: "start", baseline: "middle", opacity: 0, }); const rule = yield* Scene.instantiate("Line", { start: Entity.vec3({ x: -176, y: -110 }), end: Entity.vec3({ x: -176, y: -110 }), strokeColor: coral, strokeWidth: 6, }); for (const child of [flood, sun, gem, wordmark, tagline, rule]) { yield* Scene.appendChild(lockup, child); } // ── Bar 1 · count-in. Four ticks; the last one turns hot and // collapses into the dot the drop explodes from. for (const [i, mark] of marks.entries()) { const last = i === 3; yield* Audio.play(tracks.tick, { gain: last ? 1.8 : 1 }); yield* Scene.update(numeral, (d) => ({ ...d, text: `${i + 1}`, fillColor: last ? coral : cream, })); yield* onBeat( Motion.scale(numeral, 1.5, 1, beat * 0.36, "easeOutExpo"), Motion.tweenTo(mark, { fillColor: last ? coral : cream }, beat / 5), last ? Effect.gen(function* () { yield* Scene.sleep(beat * 0.4); yield* Scene.all([ Motion.scaleTo(numeral, 0, beat * 0.6, "easeInBack"), Motion.tween( field, { radius: 0 }, { radius: 14 }, beat * 0.6, "easeInExpo", ), ]); }) : Effect.void, ); } // ── Bars 2–3 · the drop. The groove lands on the downbeat. const groove = yield* Audio.play(tracks.groove, { loop: true }); yield* hide(count); yield* Scene.fork( Motion.tweenTo(field, { radius: 760 }, beat * 0.7, "easeOutExpo"), ); for (const [i, text] of ["EVERY", "FRAME", "ON THE", "BEAT"].entries()) { const tilt = [-0.06, 0.05, -0.035, 0][i] ?? 0; yield* Scene.update(shout, (d) => ({ ...d, text, rotation: Entity.vec3({ z: tilt }), })); const [top, bottom] = streaks; yield* onBeat( Motion.scale(shout, i === 3 ? 1.9 : 1.4, 1, beat * 0.4, "easeOutExpo"), top === undefined ? Effect.void : Motion.move( top, { x: -1500, y: 250 }, { x: 1500, y: 250 }, beat, "easeInOutQuart", ), bottom === undefined ? Effect.void : Motion.move( bottom, { x: 1500, y: -250 }, { x: -1500, y: -250 }, beat, "easeInOutQuart", ), ); } // Hard cut to black: eight columns shoot out of the center line. yield* hide(drop); yield* show(pulse); yield* show(pulseWord); const ripple = (depth: number, rise: number) => Scene.stagger( pairs.map((pair) => Scene.all( pair.map((column) => column === undefined ? Effect.void : Motion.scale( column, { x: 1, y: depth }, { x: 1, y: 1 }, rise, "easeOutExpo", ), ), ), ), Schedule.spaced(beat * 0.08), ); yield* onBeat( ripple(0, beat * 0.6), Motion.scale(pulseWord, 1.8, 1, beat * 0.5, "easeOutExpo"), ); for (let i = 0; i < 3; i++) { yield* onBeat( ripple(0.45, beat * 0.6), Motion.scale(pulseWord, 1.14, 1, beat * 0.4, "easeOutCubic"), ); } // ── Bar 4 · the sweep. The riser's length is the transition's length, // and the groove crossfades into the drift across all of it. const drift = yield* Audio.play(tracks.drift, { gain: 0, loop: true }); yield* Audio.play(tracks.riser); const sweepMs = Duration.toMillis(sweep); yield* Scene.all([ Audio.crossfade(groove, drift, sweep, "easeInOutSine"), Scene.all([ Motion.scaleTo(pulseWord, { x: 2.4, y: 0.6 }, beat, "easeInCubic"), Motion.fadeTo(pulseWord, 0, beat, "easeInCubic"), ]), Effect.gen(function* () { // squeeze (with a small anticipatory widen), fold, stretch, open yield* Motion.scaleTo(pulse, { x: 0.06 }, sweepMs * 0.5, "easeInBack"); yield* Motion.rotateTo( pulse, Math.PI / 2, sweepMs * 0.25, "easeInOutCubic", ); yield* Motion.scaleTo( pulse, { x: 0.012, y: 2.4 }, sweepMs * 0.125, "easeInExpo", ); yield* Scene.all([ Motion.scale( backdrop, { x: 1, y: 0.004 }, { x: 1, y: 1 }, sweepMs * 0.125, "easeOutExpo", ), Motion.fadeTo(pulse, 0, sweepMs * 0.125), Motion.fadeTo(tunnel, 1, sweepMs * 0.125, "easeOutCubic"), ]); }), ]); yield* Audio.stop(groove); yield* hide(pulse); // ── Bars 5–6 · drift. The camera flies through a tunnel of rings while // flat type rises over it; each beat flares the next ring. yield* show(title); const camera = yield* Scene.camera; yield* Scene.fork( Motion.moveTo(camera, { z: -1000 }, beat * 7, "easeInOutSine"), ); yield* Scene.fork( Scene.stagger( letters.map((letter) => Scene.all([ Motion.moveTo(letter, { y: -205 }, beat * 1.4, "easeOutCubic"), Motion.fadeTo(letter, 1, beat * 1.4), ]), ), Schedule.spaced(beat * 0.24), ), ); yield* Scene.fork( Effect.gen(function* () { yield* Scene.sleep(beat * 2); yield* Motion.fadeTo(caption, 0.8, beat); }), ); yield* Scene.fork( Effect.gen(function* () { yield* Scene.sleep(bar); yield* Scene.all( letters.map((letter, i) => Motion.moveTo(letter, { x: (i - 2) * 196 }, bar * 0.75), ), ); }), ); for (let i = 0; i < 7; i++) { const flare = rings[i + 3]; yield* onBeat( flare === undefined ? Effect.void : Effect.gen(function* () { yield* Motion.tweenTo( flare, { strokeWidth: 22, strokeColor: ice }, beat * 0.12, ); yield* Motion.tweenTo( flare, { strokeWidth: 5, strokeColor: cyan }, beat * 0.7, "easeOutCubic", ); }), Motion.rotateTo( diamond, Math.PI / 4 + ((i + 1) * Math.PI) / 2, beat * 0.6, "easeOutBack", ), ); } // ── Bar 7 · collapse. One beat of held breath: sound and picture pull // into a point. yield* onBeat( Audio.fadeOut(drift, beat * 0.9), Motion.moveTo(camera, { z: -1500 }, beat, "easeInCubic"), Motion.fadeTo(tunnel, 0, beat, "easeInCubic"), Motion.fadeTo(caption, 0, beat * 0.5), ...letters.map((letter) => Scene.all([ Motion.moveTo(letter, { x: 0, y: 0 }, beat, "easeInBack"), Motion.scaleTo(letter, 0, beat, "easeInBack"), ]), ), ); yield* Audio.stop(drift); // ── Bars 7–8 · lockup. The hit floods the frame and the marks from the // earlier shots return as a wordmark. The hit's length is the ending: // the picture fades out with the last beat of its chord. yield* Audio.play(tracks.hit); yield* hide(title); yield* hide(tunnel); yield* hide(backdrop); yield* show(lockup); yield* Scene.fork( Motion.tweenTo(flood, { radius: 760 }, beat * 0.6, "easeOutExpo"), ); yield* Scene.fork(Motion.scale(lockup, 1, 1.04, ring)); const after = <R>( delay: number, effect: Effect.Effect<unknown, never, R>, ) => Scene.fork( Effect.gen(function* () { yield* Scene.sleep(delay); yield* effect; }), ); yield* after(beat * 0.3, Motion.scale(sun, 0, 1, beat, "easeOutBack")); yield* after( beat * 0.5, Scene.all([ Motion.scale(gem, 0, 1, beat, "easeOutBack"), Motion.rotate( gem, -Math.PI / 2, Math.PI / 4, beat * 1.4, "easeOutCubic", ), ]), ); yield* after( beat * 0.6, Scene.all([ Motion.move( wordmark, { x: -120, y: 24 }, { x: -180, y: 24 }, beat, "easeOutExpo", ), Motion.fadeTo(wordmark, 1, beat * 0.6), ]), ); yield* after( beat * 1.1, Scene.all([ Motion.move( tagline, { x: -176, y: -80 }, { x: -176, y: -62 }, beat, "easeOutCubic", ), Motion.fadeTo(tagline, 1, beat), ]), ); yield* after( beat * 1.5, Motion.drive(rule, beat * 1.4, "easeInOutCubic", (t, d) => ({ ...d, end: Entity.vec3({ x: -176 + t * 506, y: -110 }), })), ); yield* Scene.sleep(Duration.toMillis(ring) - beat * 2); yield* Motion.fadeTo(lockup, 0, beat * 2, "easeInOutSine"); }, { width: WIDTH, height: HEIGHT, backgroundColor: ink },);