Search by what’s in the shot
Producers knew the shot they wanted. The archive only knew file names.
How do you search a video archive by describing what’s in the shot?
Turn every file into an embedding, a list of numbers that captures what’s in it, and search by meaning instead of by name. At Red Bull, producers found footage by its file name, and a clip they half-remembered could take more than 30 minutes to dig out. I tested six multimodal models for accuracy, false positives and cost, then we built search on embeddings, so a producer types what they remember seeing, such as “Max Verstappen holding Red Bull can”. Finding a clip now takes under 60 seconds.
The slow step was the file name
A file name only says what someone thought to type when they saved it. I had run keyword search before, on Elasticsearch for the Red Bull TV app, and keyword search matches words in a title. A producer looking for a shot rarely knows the title, so the search had to understand what was in the footage itself.
Six models, three measures
Before choosing anything, I tested six multimodal models: GPT-4o, Claude, Llama, Gemini, Watson and Qwen. I built the evaluation framework that scored each one on three measures:
- Accuracy: how often the model gets what’s in the shot right.
- False positives: how often it sees something that isn’t there. In search, a false positive is a wrong clip near the top of the results, and it costs a producer the time the tool is meant to save.
- Cost: what the model charges to process a file, since every file in the archive has to go through it.
The results guided which vendor and model we chose and how we deployed it.
Search by what you remember
Each video, image and audio file in the library becomes an embedding, and so does the producer’s description. The files whose embeddings sit closest to the description come back first. Nobody has to tag a file for it to be found, and nobody has to guess what it was called.
What changed
Finding a clip went from more than 30 minutes to under 60 seconds. The search sits beside the video tagging pilot I led: one has a model label what happens in a clip, the other finds the clip from a description.