Introduce yourself
Menu

Red Bull · Product consultant, emerging technologies

Teaching a model to watch breaking

A breaking battle moves faster than anyone can type.

How do you tag every move in a breaking battle with AI?

At Red Bull, a multimodal model watches the video and labels each move. A person checks only the moves the model isn’t sure of, and the finished labels reach the live broadcast in real time through an API. Tagging that used to take producers hours now takes 3 seconds, and the model is 87% accurate across more than 5,100 labelled clips.

The slow step

Producers tagged what happens in every clip by hand, and it took hours. Nothing further down the line could move faster than that, which made it the slowest step in the whole system. Industrial engineering taught me to start there, so I did.

What I built

I proposed Red Bull’s first video annotation pipeline, and I run it today. It works in three steps:

  1. A multimodal model watches the clip and labels each move.
  2. A person decides the moves the model isn’t sure of.
  3. The labels go out through an API, fast enough to reach the live broadcast.

We started with breaking, and built the pipeline so other sports can follow.

Why a person stays in the loop

The model is fast, but it isn’t always sure. Sending only its uncertain moves to a person means people spend their time where the model needs them most, and their answers train the next version. It’s also the clearest way I know to show how AI learns, which is why the demo below lets you be that person.

How we know it works

The model is 87% accurate across more than 5,100 labelled clips. I also co-wrote a paper, Breakdance Video Classification in the Age of Generative AI, comparing encoder and decoder video foundation models (ImageBind, ViViT and Qwen among them) for recognizing breaking moves. It’s under review, and I present it at INFORMS 2026.

From proposal to pilot

I took the pipeline from a proposal to a pilot: the spec, the vendors, the evaluation and the funding. Google, Microsoft, IBM and NVIDIA contributed compute and engineering.

What it’s teaching me next

Lately I’ve been thinking about what training and evaluating models on human motion in live sport can teach world models for cars and robots. I compared notes on it at the Agentic AI Summit at Berkeley RDI in August 2026.

Try it yourself

Try it: be the person in the loop

A minute of a breaking battle, cut into moves. Press play and watch the model label each move as it happens. Then decide the ones it isn’t sure of.

  1. 0:00Toprock
  2. 0:07Toprock
  3. 0:13Footwork
  4. 0:21
    Freeze?, Check
    Which move is at 0:21?
  5. 0:26Power move
  6. 0:34
    Footwork?, Check
    Which move is at 0:34?
  7. 0:43Power move
  8. 0:51
    Freeze?, Check
    Which move is at 0:51?

Illustration. An invented round drawn as a skeleton: not Red Bull footage or model output.

Back to the bookshelf