ImplicitQA: Going beyond frames towards Implicit Video Reasoning

Sirnam Swetha, Rohit Gupta, Parth Parag Kulkarni, David G Shatwell, Jeffrey A Chan Santiago, Nyle Siddiqui, Joseph Fioresi, Mubarak Shah
CRCV, University of Central Florida
arXiv Dataset Code BibTex

ImplicitQA is the first benchmark designed to test a model’s ability to perform implicit reasoning in videos - challenging systems to infer what’s not explicitly shown on screen, much like how humans naturally understand movies and TV shows. While existing VideoQA datasets tend to focus on questions that can be answered from what is directly visible in a frame or clip, ImplicitQA goes a step further. It evaluates deeper narrative understanding by requiring models to reason about hidden causality, off-screen actions, social cues, and commonsense inferences that unfold across time. Our dataset is carefully annotated by graduate researchers with expertise in vision-language models, ensuring high-quality questions that reflect the kinds of assumptions and contextual leaps humans make without even thinking. The benchmark is deliberately challenging: even state-of-the-art models like OpenAI’s GPT-O3 achieve only 64% accuracy, compared to 85% for non-expert human annotators. This significant gap highlights how far current systems are from truly understanding the subtle, implicit logic that underpins real-world video content.

Figure: ImplicitQA examples, each targeting a distinct implicit-reasoning dimension. (a) Lateral spatial reasoning - identifying the toy opposite the wizard clock by mentally mapping objects across the scene. (b) Motion and trajectory dynamics - inferring that black bullets move away from Mario by integrating actions and character positions. (c) Inferred counting - determining which animal is the third to leave a bridge by tracking sequential departures that are never fully visible onscreen. Models that excel at explicit perception often fail on these tasks, highlighting the need for benchmarks that probe deeper narrative understanding.



Abstract

Video QA has made significant strides by leveraging multimodal learning to align visual and textual modalities. However, current benchmarks overwhelmingly focus on questions answerable through explicit visual content - actions, objects & events directly observable within individual frames or short clips. In contrast, creative and cinematic videos - such as movies, TV shows, and narrative-driven content - employ storytelling techniques that deliberately omit certain depictions, requiring viewers to infer motives, causality, and relationships across discontinuous frames. Humans naturally excel at such implicit reasoning, seamlessly integrating information across time and context to construct coherent narratives. Current VideoQA systems and benchmarks fail to capture this essential dimension of human-like understanding. To bridge this gap, we present ImplicitQA, a novel benchmark specifically designed to test models on implicit reasoning. It comprises 1K meticulously annotated QA pairs derived from 320+ high-quality creative video clips, systematically categorized into key reasoning dimensions: lateral and vertical spatial reasoning, depth and proximity, viewpoint and visibility, motion and trajectory, causal and motivational reasoning, social interactions, physical context, and inferred counting. These annotations are deliberately challenging, crafted by authors ensuring high-quality. Our extensive evaluations on leading VideoQA models reveals performance degradation, underscoring their reliance on surface-level visual cues and highlighting the difficulty of implicit reasoning. Performance variations across models further illustrate the complexity and diversity of the challenges presented by ImplicitQA. By releasing both the dataset and our data collection framework, we aim to stimulate further research and development in the community.

ImplicitQA raises a new research challenge: to build models capable of deep temporal reasoning and implicit inference across frames.

Main contributions:
  • We introduce ImplicitQA, the first benchmark designed to test implicit reasoning in VideoQA, focusing on questions that require inference beyond direct visual observations.
  • We manually curate a high-quality dataset of 1k question-answer pairs across 320+ diverse video clips, with annotation conducted by experts in computer vision to ensure rigor and relevance.
  • We define a taxonomy of 9 categories, covering lateral spatial reasoning, depth and proximity, causal inference, social dynamics, and more, to facilitate targeted analysis and benchmarking.
  • ImplicitQA covers 15 diverse genres, spans 7 decades (from the 1960s to today), and includes both animated and live-action media.
  • We benchmark SoTA VideoLLMs on ImplicitQA and reveal significant performance degradation, highlighting the gap between current capabilities and true narrative understanding.

ImplicitQA Dataset Overview

Table:

Comparison of ImplicitQA with existing VideoQA datasets. ImplicitQA uniquely focuses on implicit reasoning with visual content, annotated end-to-end by domain experts.

ImplicitQA has nine categories varying across 15 genres with live-action and animation media types.

Left: Distribution across categories. Middle: ImplicitQA statistics across primary genres for the top seven most frequent categories.Right: Distribution of Media Type.



ImplicitQA Curation Pipeline

Figure: We begin by selecting creative video clips and download them. An expert-annotator pool then uses our FrameQuiz Annotation Tool to (1) mark temporal segments, (2) add a multiple-choice question and its correct answer for the segment, and (3) craft plausible distractor options. These annotated clips form the raw ImplicitQA Dataset. Next, a non-expert annotator pool employs the ImplicitEval Annotation Tool to answer each question, yielding a human baseline accuracy score. We run GPT-4.1 on the dataset to automatically assign initial category tags, which are then relabeled by the expert annotators.



Experimental Results

Performance of Open- and Closed-Source Models on ImplicitQA

Results on ImplicitQA, best and second best results are highlighted.


Impact of model scale on ImplicitQA

Left: Overall performance vs Model Scale, Right: Category-wise performance vs Model Scale.

BibTeX

@article{swetha2025implicitqa,
title={ImplicitQA: Going beyond frames towards Implicit Video Reasoning},
author={Swetha, Sirnam and Gupta, Rohit and Kulkarni, Parth Parag and Shatwell, David G and Santiago, Jeffrey A Chan and Siddiqui, Nyle and Fioresi, Joseph and Shah, Mubarak},
journal={arXiv preprint arXiv:2506.21742},
year={2025}
}