1. Introduction
The most useful thing about language models is that you can change what they do by changing what you say. We wanted the same property for live video: to be able to say "show me the brands on screen" or "build a panel with the ad categories for this moment" and have it happen over a live stream, without a new release.
The honest version of that promise has conditions. An instruction can change what the model looks for and how it answers. It cannot make the model see what is not there, and it cannot by itself create a new place for the answer to go. This report is about where those lines fall.
2. Related work
Instruction following in language models was shaped by work such as InstructGPT [1]. Using language models to compose tools and programs is explored by ReAct [2], Toolformer [3], and, for vision, VisProg [4] and ViperGPT [5], where a language model produces a program that calls visual modules. Visual instruction tuning [6] trains vision language models to follow instructions about images directly. Our system combines a vision-language perception layer with a language layer that builds apps, as described in Technical report 03.
3. The pattern
Every programmable behaviour in our system has four parts:
| Part | What it is | Example |
|---|---|---|
| Instruction | What to look for and how to choose | "Return grounded keywords and allowlisted categories; do not name individuals." |
| Evidence | The video, what is said, and any prior context | The current segment, its transcript, candidate keywords |
| Answer shape | A strict schema the answer must match | Keywords, keyword–context pairs with one category id each |
| Checks and routing | Code that validates, remembers and sends the answer somewhere | Category ids checked against the allowlist, then held for the ad system |
Change the instruction and the behaviour changes within the same answer shape. Change the answer shape and you also need new checks and somewhere for the new fields to go.
4. Three places instructions live
4.1 Built-in tasks
The enrichment task described in Technical report 02 has an instruction and answer shape defined by our engineers. Brand recognition has its own, stricter instruction and its own answer shape. These are the foundations other behaviour builds on.
4.2 Channel guidance
A channel owner can describe, in plain language, what they want: topics to favour or avoid, brands they work with, the language of the audience. A language model turns that guidance into short, specific instructions for each decision the shopping path makes: analysing the scene, choosing a product, choosing a deal, choosing between candidates. These instructions are stored with the channel's settings and applied to each relevant task, with priority over the general instructions, every time the channel's settings are refreshed. The owner changes behaviour without anyone editing code. This is the clearest case of an instruction acting as a program in our system today.
4.3 The builder interface
In the builder interface, a person points the system at a live stream and types what they want. The VLM watches the stream; the language model underneath builds the app, a panel beside the picture or an overlay on it; and the app can be saved and switched on over the live stream. In a recorded builder session on a live sports stream, instructions of this kind were typed and produced working apps. These two are as shown in our explainer film:
Build an ad enrichment panel: show the IAB categories, scene keywords, brands on screen and a brand safety level for this moment.
Now change it to brand recognition: list every brand visible on screen and where it appears.
The second instruction changed the app on the same stream without new code. Both apps read the same underlying signals; the language model built different views over them.
5. What changes with words, and what needs engineering
| Change | What it takes |
|---|---|
| Emphasise or avoid topics on a channel | Instruction only (channel guidance) |
| Prefer a list of brands on a channel | Instruction and channel settings |
| A new view over existing signals | Typed instruction |
| A different taxonomy version in the output | Supported setting, from the versions we already map |
| A new kind of output field | New answer shape, new checks, and somewhere for it to go |
| A new destination for answers | Engineering on the receiving side |
| Recognising exact content on a TV | Registration and fingerprints, not an instruction (see Technical report 01) |
6. Keeping instructions safe
Two risks come with programmable behaviour. The first is that words in the stream act as instructions. Our perception instructions treat transcripts and on-screen text as evidence, never as commands. The second is that a typed instruction asks for something the model cannot ground, such as an exact product match from a blurry shot. Our answer shapes allow "nothing found", and our apps are built to show nothing rather than guess.
7. Limitations
- Not every use is one instruction away. Enrichment, brand recognition and shopping have different instructions, answer shapes and surrounding code. A genuinely new use usually needs all four parts of the pattern.
- Shopping is shown as demos. Shopping apps built by typing detect products and scenes; connecting every app to live product catalogues is ongoing work, and we make no claims about exact-item matching.
- Recorded sessions are evidence, not benchmarks. The builder examples above show that typed instructions produced working apps on a live stream. They are not a measure of how often that succeeds.
References
- L. Ouyang et al. Training Language Models to Follow Instructions with Human Feedback. NeurIPS 2022. arXiv:2203.02155
- S. Yao et al. ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. arXiv:2210.03629
- T. Schick et al. Toolformer: Language Models Can Teach Themselves to Use Tools. NeurIPS 2023. arXiv:2302.04761
- T. Gupta and A. Kembhavi. Visual Programming: Compositional Visual Reasoning Without Training. CVPR 2023. arXiv:2211.11559
- D. Surís, S. Menon and C. Vondrick. ViperGPT: Visual Inference via Python Execution for Reasoning. ICCV 2023. arXiv:2303.08128
- H. Liu, C. Li, Q. Wu and Y. J. Lee. Visual Instruction Tuning. NeurIPS 2023. arXiv:2304.08485