MotionBlind Probing the Illusion of Motion Understanding in Video‑LLMs

Abstract

A video large language model can watch two clips of the same person in the same room and name every object in both, yet fail to tell you which one moves faster, which way a hand travels, or how hard a door is shut. That gap matters because Video‑LLMs are increasingly the perceptual front end of world models: auto‑labeling state transitions, acting as reward and success detectors for model‑based RL, grounding instructions in what an agent sees. Each of those roles presupposes a competence at reading motion that, we show, these models do not have.

MotionBlind is a contrastive, minimal‑pairs benchmark of self‑recorded clip pairs that are near‑identical except in motion, each paired with two complementary yes/no questions, so a model must answer all four items correctly (Instance Accuracy, I_Acc) to score the instance. Single‑frame, appearance and language‑only shortcuts all collapse toward the 6.25% chance floor. It complements TimeBlind with physically grounded categories (speed, magnitude, direction) that are hard to source and label from uncontrolled internet video but are precisely the variables a world model must predict.

Across two suites we run a controlled study of six open Video‑LLMs and two frontier models over frame budgets 1–24 and four frame‑selection strategies. Open models sit near the chance floor; a larger model does no better; and neither more frames nor dynamic selection, which changes which frames are seen and not whether motion is read, closes the gap. The task genuinely needs video in the right order: removing it zeroes every model, and shuffling frames collapses accuracy to chance. Only Gemini 3.1 Pro clears the benchmark overall, yet it too fails on the purely rate‑defined categories.

The contrastive design

One instance is two near‑identical clips × two opposed questions = four binary items. The clips differ only in motion; the questions are logical opposites. A model must get all four items right to score the instance.

That structure is what makes the chance floors so unforgiving, and why a headline accuracy near 50% means nothing on its own:

What the benchmark is made of

Sixty contrastive instances over 82 self‑recorded clips. Every category is physical/kinematic: exactly the axis that is hard to source and label from uncontrolled internet video, and directly controllable in a lab. All clips are self‑recorded with object, scene and background held fixed across each pair, so the contrast isolates the motion property under test; and every relative‑characteristic instance compares two moving references, so each question is well‑posed from the clips alone rather than depending on an unstated calibration of what counts as fast or hard.

Each clip pair also carries one to three question levels over identical footage: the directly visible kinematic difference (which motion is faster, farther, in which direction), the inferred dynamic cause (for appearance‑matched objects, which was pushed harder), and a chained deduction (more distance in equal time implies faster). Sharing the same clips, these levels localize a failure to perception, to inference, or to the link between them, rather than to a single conflated score.

How it differs from TimeBlind

Both suites share the same 2×2 protocol and the same yes/no format, but they ask about different things: MotionBlind's questions turn on how a motion is performed, TimeBlind's on what changes and when. That shows up directly in the vocabulary.

Results: open models fail, one frontier model breaks away

Each model's best cell across its full sweep (3–5 samplers × 5–6 frame budgets), scored on all 60 instances. These are best cells chosen post hoc, so they are optimistic by construction. Ties on I_Acc are broken toward uniform sampling and then the largest budget: the configuration that gave the model the most to work with, so a tie is never resolved in the benchmark's favour.

^∗ evaluated on a representative 56‑instance subset (224 questions); not 1:1 comparable. ^† reasoning model, evaluated with reasoning disabled to match the benchmark's no‑CoT convention. ^‡ motion‑tuned Qwen2.5‑VL‑7B. ^¶ GPT‑5.6 (Luna) at 720×1280, detail=high. ^§ TimeBlind's own published best over 20+ frontier MLLMs, not our run. ^‖ mean of 5 annotators (60 instances each, range 85–97 I_Acc); TimeBlind human from the TimeBlind paper.

Do the models read the video at all?

Four arms at a fixed budget of 16 frames. ordered, reversed and shuffled receive the identical frames: uniform sampling is deterministic, so only the sequence order changes. no‑video receives the question with no frames at all. Every contrast is exactly paired, so McNemar's exact test applies.

Reversal and shuffling are not the same probe. Shuffling destroys order; reversing inverts it, flipping the correct answer for every first/second and direction question. Scored against the original labels, a model that genuinely reads direction therefore lands below the 50% floor, so reversed is the only arm where failure is evidence of perception.

The same collapse, across every model we could run it on

All four arms at the same matched 16‑frame budget, on the four models whose ablation cells we have. The pattern is not a Qwen quirk.

All values are I_Acc (%) at uniform‑16 on all 60 instances; chance is 6.25. ^† a single instance out of 60, i.e. noise.

Scrambled frames are exactly as useful as no frames. The shuffled vs no‑video contrast returns b = 50, c = 50, p = 1.0000, a perfect tie, and both confusion matrices are exactly symmetric (24/24/96/96 and 32/32/88/88). Ordered beats shuffled at p = 0.0226. So temporal order carries real information, but not enough to lift a single instance above luck.

The reversed arm sits at 45.4% accuracy, below the 50% floor. That is the signature of a model tracking direction just well enough to be reliably wrong once the clip is played backwards.

TimeBlind starts far higher (27.0% I_Acc with real order, well above chance) and still falls to 7.0% under both reversal and shuffling, then to 0.0% with no video. MotionBlind never leaves the floor at all (6.7 → 0.0). The gap matters for what the benchmark is for: TimeBlind's event-ordering questions are partly answerable from semantics, so destroying order costs a lot but not everything. MotionBlind's contrastive pairs share every semantic cue by construction, which is why there is nothing left to lose.

Where the failure lives

Breaking I_Acc down by motion characteristic localizes the failure. It is not that the models are uniformly blind to time: they specifically fail to encode how fast and how far, exactly the kinematics a world model must predict.

More frames help, then plateau far below solving

Since the models are order‑sensitive, does simply showing more of the clip help? Only up to a point, and that point is far below competence.

GPT-5.6: the same shape at the frontier

Selection is frozen in a first stage and replayed in a second, so every cell differs only in which indices were chosen, so a re‑run is bit‑identical.

Dynamic frame selection does not rescue it

If uniform sampling saturates, perhaps a smarter selector would spend the budget better. We compare uniform and random against a GRPO‑trained learned selector (HORNet) and a training‑free key‑clip selector (Frame2Clip), all at a matched 16‑frame budget.

The learned selector offers no lever here for a diagnosable reason, and the rest of this section is that diagnosis.

Why the selector has nothing to select on

Each clip contains two episodes of the same action at different speeds, and the question compares them. Frame‑differencing separates the two halves cleanly: mean absolute difference between consecutive frames is 1.6–1.7× higher in the fast half, and recovers the gold answer on both clips. The signal is present in the pixels.

What the model actually receives. Ten instances spanning all four motion characteristics. Pick one: the two clips are near‑identical and the gold answers are opposite, so any strip you cannot tell apart is one the model cannot either.

A stirring cycle is 15–30 frames. Uniform sampling steps over two to three complete cycles between samples: the classic below‑Nyquist case, where a fast cycle and a slow cycle produce identical sample sequences. Across all three samplers there is almost nowhere that two frames land close enough together to measure a rate.

This makes the failure specific: it is a sampling failure, not a signal failure. Samplers are built to maximise coverage, which is precisely the wrong objective for a question about rate.

What this does and does not show

MotionBlind's 60 self‑recorded instances trade scale for controlled isolation of physical variables, so every number here carries small‑sample variance: one instance is 1.7 points of I_Acc, and 14.3 points inside the 7‑instance speed category. All clips feature a single actor indoors, leaving in‑the‑wild and egocentric generalization open. Constrained yes/no parsing removes free‑form ambiguity but may understate reasoning a model cannot verbalize in binary form. Our open‑weight models span ≤12B parameters, so we cannot fully separate architectural from scale limits, though a larger model not closing the gap is suggestive.

All video was collected with the participant's informed consent, faces are automatically blurred prior to release, and the dataset contains no sensitive or personally identifiable information.

Reproducing

The dataset is on the Hugging Face Hub:

from datasets import load_dataset

ds = load_dataset("augmentedcognitionlab/MotionBlind")

BibTeX

To be added.