Perception Before Reasoning: Dynamic Latent Reasoning for Video Understanding and Question Answering
News Release Summary
Video question-answering systems often produce long written reasoning traces even when a question can be answered by locating the right moment in a clip. This paper introduces Dynamic Latent Reasoning (DyLaR), which represents visual evidence and, when needed, additional reasoning as compact internal latent states rather than lengthy text. The system learns when to stop after perception and when to add a reasoning step. Across nine benchmarks and four model backbones, the authors report higher average accuracy with fewer than 20 generated tokens per question; on Qwen3-VL-4B, DyLaR raises average accuracy from 54.0 to 58.2 while reducing average response length from 1,220.7 to 18.5 tokens.
abstract
Video question answering requires models to ground language queries in visual evidence and, when necessary, reason over that evidence across time. Existing methods typically rely on long textual chain-of-thought rationales, even though many questions can be answered as soon as the relevant object, action, or frame is localized. We propose Dynamic Latent Reasoning (DyLaR), which first grounds a question in a short block of perception latents (continuous hidden states that encode query-relevant visual evidence), and then adaptively decides whether to append reasoning latents (continuous thoughts that reason over this evidence in latent space) before answering. DyLaR learns this behavior by grounding perception latents in verified visual evidence and distilling verified rationales into reasoning latents, followed by reinforcement learning that further refines when to reason. Across nine video benchmarks and four multimodal language model backbones, DyLaR improves average accuracy over same-backbone baselines while generating fewer than 20 tokens per query. On Qwen3-VL-4B, for example, DyLaR improves average accuracy over Qwen3-VL-4B-Thinking from 54.0 to 58.2 while reducing response length from 1,220.7 to 18.5 tokens per query. Ablations further show that grounded perception latents, rationale-supervised reasoning latents, and adaptive routing each improve accuracy.
citation
@article{xiaperception,
title = {Perception Before Reasoning: Dynamic Latent Reasoning for Video Understanding and Question Answering},
author = {Xia, Haotian and Xiao, Zilin and Zou, Junbo and Ordonez, Vicente and Chen, Hanjie},
journal = {arXiv preprint arXiv:2608.04124},
url = {https://arxiv.org/abs/2608.04124},
}
automatically generated questions, main contributions and limitations of this paper
Questions this paper helps answer
- What problem does DyLaR address? DyLaR reduces the need for long textual reasoning traces in video question answering by first grounding a question in compact visual-evidence latents and adding reasoning latents only when needed.
- How does DyLaR decide whether to reason further? It learns an adaptive routing behavior that can answer after perception or append latent reasoning before producing an answer.
- What supervision is used to train the latent representations? The method grounds perception latents in verified visual evidence, distills verified rationales into reasoning latents, and then refines the behavior with reinforcement learning.
- What results are reported for DyLaR? Across nine video benchmarks and four backbones, the paper reports improved average accuracy with fewer than 20 output tokens per query; Qwen3-VL-4B improves from 54.0 to 58.2 average accuracy against its thinking baseline.
Main contributions
- DyLaR separates query-relevant visual perception from optional latent reasoning, allowing video QA to use different amounts of computation for different questions.
- The training recipe combines evidence-grounded perception supervision, rationale-derived reasoning supervision, and reinforcement learning for adaptive routing.
- Ablations reported in the paper find gains from grounded perception latents, rationale-supervised reasoning latents, and the adaptive routing component.
- The evaluation spans nine video benchmarks and four multimodal language-model backbones.
Limitations and cautions
- The current evaluation focuses on video question answering, so applying the same routing strategy to other multimodal tasks remains a useful next step.
- The method relies on verified visual evidence and verified rationales during training; future work could examine how readily these supervision signals scale to new video domains.
- The reported efficiency measure emphasizes generated-token length, while broader deployment studies could also measure latency, memory use, and visual-token compute.
- The results are benchmark-based, but the consistent improvements across multiple backbones provide encouraging evidence that latent routing is a practical direction.
How to read this result
DyLaR is best read as a well-supported approach to making video QA more selective about when it performs explicit reasoning, with strong benchmark evidence and clear room to test its efficiency benefits in broader settings.