klob/research klob.io

Machine learning

Neural rhythm tracking

Compare recurrent networks, particle filters, transformers, and phrase-specific models.

A neural beat tracker learns which combinations of timbre, rhythm, and musical context predict a labeled event. Its output might be one tempo class per excerpt, per-frame beat and downbeat activations, or named functional sections. The decoder that turns these outputs into events can change the behavior as much as the network itself. 22 · 21 · 26 · 19

From features to decisions

A typical front end maps a waveform to a time–frequency representation, often a log-mel spectrogram. A convolutional network learns local patterns across time and frequency. A recurrent network carries context from frame to frame. A bidirectional network can use sounds on both sides of the current frame. Transformers learn which other frames or frequency bands are useful, with their effective context set by masks, windows, and architecture. 21 · 25 · 26

A model trained on frame-level beat labels produces an activation curve. Peak picking is inexpensive but can admit extra events or uneven spacing. A DBN adds constraints on tempo and meter. Particle filtering is another option for updating several rhythmic hypotheses as the audio arrives. Learned anticipation shifts the training targets so an event can be predicted early enough for a bounded peak picker. Throughput faster than playback tells you little about future context or decision latency. 21 · 23 · 27

What the main model families return

Model What it learns and returns Where it fits
Tempo-CNN A tempo class from an 11.9 s mel excerpt. Sliding it yields a tempo curve. A BPM estimate or candidate prior. Beat times require another stage. 22
madmom joint tracker Bidirectional recurrent beat/downbeat activations followed by a meter-aware DBN. An offline reference grid. The classic joint path uses future context. 21
BeatNet / BeatNet+ Causal convolutional/recurrent evidence with sequential inference. BeatNet uses two particle filters and can infer meter. BeatNet+ adds training on drum-reduced material. First candidates to test on a stream, including a breakdown where percussion recedes. The published implementation documents microphone input. 23 · 24
Beat Transformer Dilated self-attention across time and demixed instrument channels. File analysis where separated bass, drums, and harmonic parts can clarify accents. Budget for separation and context. 25
Beat This! Convolutions and frequency/time attention, a timing-tolerant loss, and direct event extraction without a mandatory DBN. Offline file grids. The authors report stronger F1 alongside weaker continuity on some measures. 26
Beat This! Live A causal adaptation with bounded temporal context and targets shifted to anticipate events. A streaming research candidate. Its repository emphasizes training and evaluation. Measure end-to-end latency in the target runtime. 27
All-In-One Joint beat/downbeat activations and functional boundaries and labels from separated spectrograms with neighborhood attention. Pre-analysis of an arrangement. A chorus label does not enumerate the melodic phrases inside it. 19 · 18

A comparison with a stated protocol

On GTZAN, the Beat This! Live authors report online beat/downbeat F1 of 86.7 / 74.1, compared with 80.6 / 56.5 for BeatNet+ in their comparison. Downbeats are the harder target on this test. The result motivates a side-by-side trial on live mixes, especially one that records cold-start lock, stable phase through a breakdown, and the delay between prediction and a displayed frame. These are published scores, independent of this guide’s synthetic teaching loop. 27

Open code, weights, and deployment

The Beat This! and Beat This! Live repositories specify MIT for code and released weights. madmom’s repository gives its code a BSD license and distributes its model/data files under CC BY-NC-SA 4.0 by default. BTrack’s C++ repository uses GPL-3.0. These distinctions matter when choosing components to redistribute with a visualizer. For a web prototype, a local native or Python analysis service can expose timestamped events to the renderer. Revalidate output parity and timing before swapping in a browser model or quantized weights. 26 · 27 · 21 · 8

Chapter sources (10)