klob/research klob.io

Live implementation

Driving a visualizer

Connect audio features and transport clocks to a renderer with predictable timing.

For a 4/4 VJ rig, start with the clock the source can actually provide. A prepared file allows whole-song analysis and human corrections. A live DJ transport can share timing even during silence. An unknown stereo feed has only the samples received so far. The renderer can consume the same event format for each source while preserving how far ahead it can trust each event. 32 · 9 · 19

Choose the path that matches the input

Input Suggested analysis path Operator control
Music on Windows, including system playback Capture the selected output with WASAPI loopback in a native companion, or use browser display capture when the selected surface offers audio. Check that an audio track was returned. Compare real-time PLP and BeatNet on the same feed. Tap tempo, half/double, set “one,” and mark phrase start. 35 · 9 · 23
Traktor with Link Use Link for tempo and phase, with a compatible bridge for a browser renderer. Observe drift and align the deck’s beatgrid. Use the audio path for timbre, attacks, and confirmation. Reset the downbeat independently and arm transitions. 32 · 33
Loaded audio file Precompute beats/downbeats with Beat This!, compare the grid with a classical tracker, and propose sections with All-In-One. Save corrected cues keyed to the exact audio version. Scrub to the drop, move a marker, or accept and label a phrase. 26 · 7 · 19
Compatible prepared DJ hardware Read existing track and phrase data where available. Beat Link Trigger illustrates using that information for start, body, fill, and end cues. Give hand-authored track cues priority. 34

Essentia.js exposes audio algorithms through WebAssembly. Inspect each algorithm’s input requirements. Essentia’s RhythmExtractor2013 uses full-track statistics and is recommended for offline work in its own tutorial. A streaming API that accepts chunks does not by itself make the resulting beat decisions causal. For lightweight in-browser continuous features, Meyda is a useful starting point. 28 · 29

Map each signal to the right visual timescale

Audio or clock feature A concrete visual mapping Behavior to test
Smoothed RMS and band energies Bass → shape size, midrange → layer density, treble → fine detail. Tune attack and release separately. Loud masters, quiet breakdowns, and gain steps should remain legible. 29 · 30
Spectral flux and centroid Use transients to accent line weight. Use spectral brightness for a slow change in texture. A filter sweep can brighten continuously without extra beat triggers. 1 · 29
Continuous beat phase Derive a periodic motion curve and put its apex at the predicted beat. Motion should cross a brief missing kick smoothly. 9
Downbeat and group position Rotate a palette per confirmed bar or morph a shape across eight manually anchored bars. A phase correction should avoid duplicate cues. 32 · 11
Predicted or confirmed section cue Arm a scene change, then commit on a chosen downbeat. Use observed changes for effects that can arrive later. Count false hard cuts during fills, crossfades, and long drops. 31 · 17

projectM’s MilkDrop-compatible preset system exposes quick and damped bass, midrange, and treble responses. Butterchurn is a browser implementation in this family. Both are useful references for the visual side of a system even if a separate component supplies the musical clock. 30

Schedule on the audio timeline

Timestamp analysis frames in audio sample time. A useful event includes its musical time, its emission time, event kind, confidence or stability, and the transport generation that produced it. For a known file, use the playback position and rate to map cue times onto presentation time. On a seek or loop, increment the generation and cancel pending cues from the old one. A causal system can extrapolate the next beat and revise future predictions as evidence arrives. 36 · 9

In the browser, keep audio-thread work short. An AudioWorklet can gather timestamped feature frames while a worker or native process runs heavier inference. Schedule visual updates with the display’s animation callback and evaluate phase for the expected presentation time. Web Audio’s getOutputTimestamp pairs an output-context time with performance time. It does not measure input capture, a projector’s processing, or the path from a separate speaker to the audience. Measure and store a route-specific offset. 36

For example, at 60 Hz an entire display frame takes about 16.7 ms. Measure the signed difference between an audible transient and a co-recorded on-screen flash. Use a shared recorder and account for its own audio/video offset, or use a light sensor plus audio loopback for finer measurements. Correct fixed delay separately from jitter. Confidence can fall independently for the beat, bar, or phrase, so a doubtful phrase prediction need not stop the beat animation.

Chapter sources (17)