论文
arXiv
SpatialIntelligence
Trajectory
Mobility
LLM
中文标题
FastBench:流式视觉语言模型能否感知高动态真实世界视频流?
English Title
FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams?
Yuxuan Hu, Weikang Shi, Yang Bo, Xudong Lu, Xintong Guo, Shuhan Li, Yuyang He, Huankang Guan, Peiwen Sun, Yunqiao Yang, Wenbo Li, Rui Liu, Hongsheng Li
发布时间
2026/10/9 01:55:20
来源类型
preprint
语言
en
摘要
中文对照

流式视频大语言模型(VLMs)支持连续视频理解,但现有基准测试主要聚焦于低动态场景。在有限的上下文预算下,模型必须在时间历史、空间分辨率和时间粒度之间进行权衡;1-2 FPS 的稀疏采样会遗漏快速事件。我们提出 FastBench 以评估真实世界视频流中的高动态感知能力。其基于轨迹的流程结合了从高帧率片段生成问答、过滤可在 2 FPS 下回答的问题、使用 SAM3 和 CoTracker3 轨迹验证答案以及三轮人工检查。FastBench 包含 306 个问答对,涵盖八个领域、六种能力以及前向、即时和后向时间范围,并附有标注的证据区间。我们还提出了 ProactiveFrame,这是一种通过文本令牌调整输入帧率的免训练基线方法。双层级滑动窗口保留近期的高帧率观测数据,同时将较早的数据下采样为稀疏历史。实验揭示了显著的局限性:表现最强的模型 Gemini-3.5-Flash 仅得 50.7% 分。更密集的采样使 Qwen3-VL-8B 从 2 FPS 时的 32.9% 提升至 24 FPS 时的 44.6%,但随着历史数据的压缩,增益趋于饱和。ProactiveFrame 比稀疏均匀采样高出 5.4 和 1.5 个百分点,但仍远低于 oracle 引导的关注机制,这表明当前 VLMs 难以仅凭视频流判断何时需要更精细的时间感知。FastBench 为高动态流式视频理解提供了测试平台。代码和数据:https://github.com/Ashone3/FastBench。

English Original

Streaming Video Large Language Models (VLMs) enable continuous video understanding, yet existing benchmarks focus on low-dynamic scenarios. Under bounded context budgets, models must balance temporal history, spatial resolution, and temporal granularity; sparse sampling at 1--2 FPS misses fast events. We introduce FastBench to evaluate high-dynamic perception in real-world video streams. Its trajectory-grounded pipeline combines QA generation from high-FPS clips, filtering of questions answerable at 2 FPS, answer verification using SAM3 and CoTracker3 trajectories, and three rounds of human inspection. FastBench contains 306 QA pairs across eight domains, six capabilities, and forward, instant, and backward temporal scopes, with human-annotated evidence intervals. We also present ProactiveFrame, a training-free baseline that adjusts incoming frame rates through text tokens. A dual-tier sliding window retains recent high-FPS observations while downsampling older ones into sparse history. Experiments reveal substantial limitations: the strongest model, Gemini-3.5-Flash, scores only 50.7%. Denser sampling improves Qwen3-VL-8B from 32.9% at 2 FPS to 44.6% at 24 FPS, but gains saturate as history is compressed. ProactiveFrame outperforms sparse uniform sampling by 5.4 and 1.5 percentage points, yet remains well below oracle-guided focusing, showing that current VLMs struggle to determine from the stream alone when finer temporal perception is needed. FastBench provides a testbed for high-dynamic streaming video understanding. Code and data: https://github.com/Ashone3/FastBench.

我的阅读记录

正在加载阅读记录…

元数据
arXiv2610.12427v1
来源arXiv
类型论文
抽取状态raw
关键词
SpatialIntelligence
Trajectory
Mobility
LLM
cs.CV
cs.CL