klob/research klob.io

Evaluation

Measuring reliability

Test timing, continuity, startup, and false scene changes on the right annotations.

“Reliable” needs a target and a deadline. A BPM classifier, a pulse tracker, and a system that triggers the first bar of a drop can all be scored on the same song and give very different answers. Report musical accuracy and the delay at which each answer was available. 37 · 38 · 27

Match the reference to the question

Reference What it can test What requires additional labels
GiantSteps Tempo Tempo on 664 electronic music previews, including revised reference values. Beat placement, downbeats, and phrase starts. 39
Harmonix Set Beats, metrical positions, and functional sections on 912 popular tracks. Every internal phrase of a section or the precise cue a VJ would choose. 40
Raveform EDM structure including drops and breakdowns, with 1,423 structure-annotated individual tracks and beat/downbeat information. An arbitrary DJ’s live transition or every phrase in the mix. 17
FaiSonic MMD Explicitly separate phrase and section boundaries in a 500-song pop corpus. Generalization to live EDM or the performer’s preferred mixing points. 18
A locally annotated set recording Lock and recovery across actual crossfades, transport jumps, silent passages, and the display chain. Reproducible conclusions until the annotation policy, hardware, and sample are described.

Use metrics that expose the failure

Tempo. Report absolute BPM error and accuracy at a stated relative tolerance. A common Accuracy 1 convention uses 4%. Accuracy 2 also accepts metrical alternatives such as double, half, triple, and one third. Publish both and count octave errors separately. A display reading 62 on a 124 BPM track has very different consequences for motion depending on which level the performer wanted. 37

Beats and downbeats. Match event times one-to-one against their own reference lists. Report precision, recall, and F1 at a stated window. The common mir_eval beat default is ±70 ms. A system that is consistently 60 ms late can still match every beat. Report signed bias, median absolute error, and the 95th percentile absolute error for matched events, along with misses. CMLc measures the longest uninterrupted correct-level run. CMLt credits all correct-level spans. AML variants allow alternative metrical interpretations. Measure downbeat F1 separately. 38

Phrases and sections. Use matching at a tight tolerance suited to the cue and also report conventional structural windows such as ±0.5 s and ±3 s for comparison. Score the nominated downbeat or bar index for DJ grouping. Score a melodic pickup at its actual timestamp when that is the target. For labeled sections, evaluate labels as well as boundaries. Record false hard cuts per hour and how often predictions arrive before their cue deadlines. 38 · 18 · 17

A repeatable live test

  1. Choose held-out tracks and mixes. Keep tracks from the same recording, remix family, or performance out of both tuning and test sets. Document meter, tempo range, style, and what counts as a phrase.
  2. Have two listeners mark beats, downbeats, and the specific VJ cues. Retain disagreements and accept multiple defensible starts where appropriate. Distinguish measured annotations from a grid imported from software under test.
  3. Feed the same PCM to each candidate in timestamped chunks at playback speed. Log the audio timestamp, wall-clock decision time, intended visual time, and frames actually presented. For prepared-file runs, label whole-track access explicitly.
  4. Start the stream mid-bar, remove the kick, change tempo, introduce a two-bar extension, crossfade two decks, then seek. Record cold-start time, longest wrong-phase run, reacquisition time, false doubles, and missed cues. Common beat evaluations trim the first five seconds. Keep an untrimmed startup result. 38
  5. Compare manual anchoring, a classical baseline, and a causal neural tracker on the same material. Report per-track results and worst cases as well as averages. Log CPU time and missed render deadlines on the machine driving the projector.

For a first klobvision comparison, use real-time PLP and BeatNet on an identical live feed, Beat This! on the corresponding files, and a manually checked cue sheet for phrase starts. Add Beat This! Live as a third online candidate. Compare how long each stays aligned through the first drumless break and how many scene changes the performer would have reversed. This is an evaluation design. The published model comparisons and their datasets are identified in the neural chapter. 9 · 23 · 26 · 27

Chapter sources (10)