Signals and timing
Rhythm in layers
Read the same waveform as a sequence of sounds, a pulse, a bar, and a musical phrase.
A repeated kick can tell you when to brighten a shape. A visual rotation that completes on every bar needs a clock that keeps moving through a missing kick. A scene change on the next drop needs information about the arrangement. Those three effects can look at exactly the same recording and need different answers. 1 · 11 · 17
Five things a listener can hear
| Target | What to return | Example at 124 BPM in 4/4 |
|---|---|---|
| Onset | A timestamp and strength for an acoustic attack | An offbeat hi-hat arrives about 242 ms after the kick. |
| Tempo | The rate of a chosen pulse, globally or over time | 124 quarter-note beats per minute. |
| Beat | A timestamped sequence, or a continuously advancing phase | One quarter note every 483.87 ms, including silent pulses. |
| Downbeat | The beat that opens each measure | Every fourth beat, once bar phase is established, about 1.94 s apart. |
| Phrase or section boundary | A timestamp, musical level, and ideally a cue type | If the arrangement groups eight bars together, those group starts are 15.48 s apart. |
A melodic phrase can start with a pickup before beat one. A functional section such as a chorus or drop may contain several phrases. A DJ cue marks a useful performance choice and can be placed before either. Decide which kind of timestamp a visual expects before choosing a detector or reference dataset. 18 · 17 · 20
Read the onset envelope
The cheapest detector measures short-time energy or root-mean-square amplitude. Compare each frame with a running baseline, trigger on a rise, and wait briefly before accepting another event. A low-frequency version can follow a kick-heavy mix. A sustained synth raises the baseline, a snare roll produces many triggers, and a silent beat leaves no peak. As a visual input it still works well for a momentary pulse or a change in glow. 1 · 30
A more selective detector windows the samples and takes a short-time Fourier transform (STFT). Compare adjacent magnitude spectra and sum their positive changes. This spectral flux can expose a new hi-hat while the bass stays loud. Use log compression so loud bins dominate less, normalize recent history, and pick local peaks above an adaptive threshold. Separating bass, middle, and treble lets a kick and a vocal consonant carry different weights. 1 · 2
Complex-domain detectors also compare observed Fourier coefficients with predictions from recent magnitude and phase. SuperFlux takes a different route, allowing energy to move between nearby frequency bins when comparing frames. This suppresses false onsets from vibrato. A bowed note with a slow attack can still have an uncertain onset time even for a human annotator. 1 · 2
Choose an analysis clock
An illustrative browser configuration uses 48 kHz audio, a 1,024-sample window and a 256-sample hop. Each window covers 21.33 ms and a new feature arrives every 5.33 ms. A centered window needs about 10.67 ms of future samples for the timestamp at its center. A trailing window is causal, but its evidence still takes time to accumulate. Feature timestamps, decision times, and the time a pixel lights up are different quantities. The values here are design examples. 36 · 1
Chapter sources (8)
- [01]A tutorial on onset detection in music signals
- [11]A spectral difference approach to downbeat extraction in musical audio
- [17]Raveform
- [18]FaiSonic MMD
- [20]Cue point estimation using object detection
- [30]projectM preset authoring guide
- [02]Maximum filter vibrato suppression for onset detection
- [36]Web Audio timing and AudioWorklet