Teaching a model to watch breaking

How do you tag every move in a breaking battle with AI?

At Red Bull, a multimodal model watches the video and labels each move. A person checks only the moves the model isn’t sure of, and the labels can reach a live broadcast through an API. In our pilot on the Red Bull BC One livestream, tagging that took producers hours took about 3 seconds, at 87% accuracy across more than 5,100 labelled clips.

The slow step

Producers tagged what happens in every clip by hand, and it took hours. Nothing further down the line could move faster than that, which made it the slowest step in the whole system. Industrial engineering taught me to start there, so I did.

What I built

I led Red Bull’s first video annotation pipeline. It started with data design: I built a movement ontology with professional breakers, a shared structure for how moves are named and related, then designed the annotation pipeline around it and led the research team that fine-tuned a video language model. It works in three steps:

A multimodal model watches the clip and labels each move.
A person decides the moves the model isn’t sure of.
The labels go out through an API, fast enough for a live broadcast.

We started with breaking, and designed the pipeline so other sports could follow.

Why a person stays in the loop

The model is fast, but it isn’t always sure. Sending only its uncertain moves to a person means people spend their time where the model needs them most, and their answers train the next version. It’s also the clearest way I know to show how AI learns, which is why the demo below lets you be that person.

How we know it works

In the pilot, the model was 87% accurate across more than 5,100 labelled clips. I also co-wrote a paper, Breakdance Video Classification in the Age of Generative AI, comparing encoder and decoder video foundation models (ImageBind, ViViT and Qwen among them) for recognizing breaking moves. It’s under review, and I present it at INFORMS 2026.

From proposal to pilot

I took the pipeline from a proposal to a pilot: the ontology, the data, the evaluation, the partners and the funding. Applied research teams from Google, Microsoft, IBM, Twelve Labs and Eluvio AI worked with us, and NVIDIA hardware processed camera data on site.

Why we stopped

The pilot worked on a livestream watched around the world. Then we hit a research ceiling: recognizing more kinds of movement could not yet be done at a cost the business could justify. Stopping was the right call, and it taught me more than shipping would have. It is also why I want to work closer to academia: industry is reaching limits it cannot research its way past alone.

What it’s teaching me next

Lately I’ve been thinking about what training and evaluating models on human motion in live sport can teach world models for cars and robots. I compared notes on it at the Agentic AI Summit at Berkeley RDI in August 2026.