MemInStream: Memorizing or Merely Recognizing?

Rethinking Streaming Video Memory Evaluation

Zhenqi He1Dunyuan Xu2Yang Bo3,†Kien T. Pham1Yanghao Wang1Shuhan Li3Xintong Guo3Yuyang He3Fang Liu4HuanKang Guan3Jingyan Chen3Rui Liu3Long Chen1,*
1 HKUST2 CUHK3 Huawei HK4 CityU

† Project Leader · * Corresponding Author

Abstract

Streaming video understanding aims to continuously process incoming visual observations, making the retention and retrieval of relevant information increasingly important as the video history grows. Although existing benchmarks include memory-related questions, question-answering accuracy alone does not reliably measure memory capabilities. Incorrect predictions may arise from failures in visual understanding rather than forgetting, whereas correct predictions may rely on linguistic priors or recent observations without requiring historical memory. To address these ambiguities, we introduce MemInStream, a benchmark dedicated to evaluating memory capabilities in streaming video understanding. We develop an evidence-controlled validation protocol that associates each question with temporally localized evidence intervals. The protocol validates evidence solvability and memory necessity by retaining questions that models can answer from the annotated evidence but cannot resolve using recent observations, post-evidence observations, or textual input alone. This helps disentangle memory limitations from understanding failures while reducing memory-independent shortcuts. MemInStream provides a systematic evaluation of memory capabilities through four major categories, covering Episodic Retrieval, State Tracking & Updating, Multi-Evidence Integration, and Contextual Memory across diverse video scenarios. Overall, MemInStream provides a comprehensive and reliable benchmark for systematically evaluating memory capabilities in streaming video understanding.

Motivation

Figure 1. Unverified memory solvability, unverified memory necessity, and the MemInStream evidence-controlled evaluation protocol.
Figure 1. Why memory evaluation needs both evidence solvability and memory necessity. Vector PDF ↗
  • Memory is essential for streaming video understanding.

    As a video unfolds, relevant observations move into the past. Models must retain, update, and retrieve this information to answer later questions, even when it is no longer visible in recent frames.

  • Existing evaluations can admit shortcuts.

    Although many benchmarks include memory-related questions, some can be answered using recent frames, textual priors, or answer-option cues. Correct predictions therefore do not necessarily demonstrate retrieval of historical evidence.

  • Incorrect answers do not always mean forgetting.

    A model may fail even when the supporting evidence is directly available. Verifying memory solvability and memory necessity helps separate memory limitations from understanding failures and shortcut-based success.

Benchmark Construction

Candidate questions from five benchmarks and newly collected videos undergo evidence annotation, automatic validation, and human verification.
Construction pipeline. Candidate collection, question generation, evidence annotation, and quality validation. Vector PDF ↗
  1. Collect & generate

    Collect candidates from StreamingBench, OVO-Bench, TeleEgo, PhoStream, and S-EMBER. Retrieve additional videos and generate questions for broader task and scenario coverage.

  2. Annotate evidence

    Assign each question a query timestamp and the historical evidence intervals needed to answer it. Add timestamped dialogue for contextual memory tasks.

  3. Validate memory

    Use two validation models to check evidence solvability and reject questions answerable from text alone or observations after the evidence.

  4. Verify with humans

    Review answer correctness, temporal alignment, and evidence completeness against the original videos. Exclude ambiguous or inconsistent questions.

Statistics

Questions
3,799
Evidence intervals
8,217
Memory categories
4
Fine-grained tasks
17
Four memory categories and 17 tasks, with distributions of evidence durations, evidence-to-query distances, and evidence interval counts.
Benchmark statistics. Memory taxonomy and the temporal characteristics of annotated evidence. Vector PDF ↗
Questions with multiple evidence intervals
56.7%
Mean evidence interval duration
9.98 seconds
Median latest / earliest evidence-to-query distance
36 / 96 seconds
6 tasks

Episodic Retrieval

Recall facts, entities, and events from specific past observations.

Explore tasks
  • Entity / Object Recall
  • Attribute Recall
  • Action / Event Recall
  • Spatial / Location Recall
  • Temporal Fact Recall
  • Occurrence Verification
4 tasks

State Tracking & Updating

Maintain and revise information as a video unfolds.

Explore tasks
  • Latest-State Retrieval
  • State Transition Tracking
  • Identity & State Persistence
  • Correction / Revision Memory
5 tasks

Multi-Evidence Integration

Connect temporally separated observations to answer a question.

Explore tasks
  • Temporal Ordering
  • Sequential Action Integration
  • Counting
  • Cross-Time Comparison
  • Multi-hop Memory Binding
2 tasks

Contextual Memory

Combine remembered dialogue with relevant visual evidence.

Explore tasks
  • Entity–Identity Binding
  • Conversational Context Memory

Results

Exact-match accuracy (%)

A prediction is correct only when the selected answer set exactly matches the ground truth. Category scores are macro-averages over their constituent tasks.

22 configurations · Category averages

Download full results

Swipe horizontally to compare scores →

MemInStream results. Accuracy in percent. Offline input is measured in recent frames; online input in frames per second.
ModelSizeInputEpisodic RetrievalState Tracking & UpdatingMulti-Evidence IntegrationContextual Memory
Proprietary models
Gemini-3.8-Flash—4frames26.423.821.136.9
Gemini-3.8-Flash—30frames45.032.927.334.7
Doubao Seed-2.1-pro—4frames20.923.218.132.1
Doubao Seed-2.1-pro—30frames38.631.925.136.2
Open-source offline models
Qwen2.5-VL7B4frames5.15.46.715.1
Qwen2.5-VL7B30frames7.15.55.814.8
Qwen3-VL8B4frames10.29.811.420.4
Qwen3-VL8B30frames17.411.511.219.7
InternVL3.58B4frames13.121.316.533.7
InternVL3.58B30frames19.620.715.727.6
Online models · Training-free
FluxMem (Qwen2.5)7B1fps9.53.73.710.6
FluxMem (Qwen3)8B1fps28.225.218.625.9
Response-G1 (Qwen2.5)7B0.25–1†fps5.53.83.93.4
Response-G1 (Qwen3)8B0.25–1†fps32.532.920.128.0
HERMES (Qwen2.5)7B0.5fps7.85.73.410.1
Online models · Training-based
Flash-VStream (Qwen2)7B1fps26.536.028.835.1
StreamForest (Qwen2)7B1fps42.037.137.335.4
TimeChat-Online7B0.2–1*fps17.020.516.118.6
AURA8B2‡fps30.427.822.630.3
JoyAI-VL-Interaction8B2‡fps41.043.932.629.7
MOSS-VL-Realtime11B2‡fps18.214.210.914.2
MOSS-VL-Realtime11B1§fps25.327.226.624.4

Input settings. Proprietary and offline models receive only the N most recent sampled frames before the query. Online models process observations chronologically; their input column reports the sampling rate in fps.

† Response-G1 dynamically adjusts between 0.25 and 1 fps. ‡ Up to 60 recent frames per query. * TimeChat-Online uses 4–316 input frames with adaptive sampling at 0.2, 0.5, or 1 fps. § Up to 768 input frames.

Figure preview

Open full-size image ↗