Ablate-to-Validate:
Are Vision-Language Models Really Using Visual Reasoning Tokens?

Video Explanation

Abstract

Vision-language models (VLMs) are increasingly augmented with continuous or latent non-textual tokens intended to support “visual reasoning.” Despite the improved task accuracy, this alone does not show that models actually use these tokens for reasoning. Gains may instead arise from confounds such as added context length, special-token anchoring, or training-time regularization. We formalize a diagnostic principle, Ablate-to-Validate, for testing whether latent-token content is genuinely utilized. We instantiate this principle as the Token Replacement Test (TRT), a standardized suite of content-replacement ablations. TRT measures whether performance depends on the information carried by latent tokens rather than their mere presence. It probes (1) span-position bias through zero and random replacement, (2) disentangles token-budget from token-diversity effects through first-repeat and count-matched variants, and (3) evaluates information content using oracle or ground-truth token injection together with distribution-matched random baselines. As a controlled testbed, we study relative depth reasoning, where continuous depth embeddings can be inserted at explicit span positions under a fixed token budget. We train LLaVA and Qwen2.5-VL models both to predict and to consume these tokens, and show that TRT applies across heterogeneous latent-token model backbones. Concretely, we cover two trained backbones (LLaVA-13B, Qwen2.5-VL-3B) with continuous and discrete depth spans across three frozen visual encoders (SigLIP2, CLIP, DINOv2) and multiple token budgets, and additionally apply TRT to three off-the-shelf visual reasoning systems (Mirage, Mull-Tokens, CoVT) evaluated on BLINK, VSP, and CV-Bench. Our results show that accuracy gains can be a misleading proxy for latent-token reasoning: across multiple model backbones, types of continuous visual tokens, and compute budgets, VLMs retain most of the improvement even when latent-token content is corrupted or replaced, revealing a persistent gap between “having a latent channel” and actually using it as an information bottleneck. By separating true content utilization from alternative explanations, TRT provides a simple and standardized way to evaluate visual reasoning tokens in vision-language models, and we recommend reporting such diagnostics as standard practice.

Method Overview

Pipeline walkthrough

1. VLMs + Visual Reasoning Tokens

A vision-language model answers visual reasoning questions while producing an intermediate visual reasoning token span.

2. Token Replacement Test

Keep the prompt and image fixed, then replace only the visual reasoning token content to isolate what the model actually consumes.

3. Utilization Diagnosis

If content matters, corruption should hurt and oracle tokens should help. If performance is stable, tokens may be acting as anchors or budget.

Datasets

The training and evaluation data are released publicly on the Hugging Face Hub.

Key Results

Beyond Depth — Off-the-Shelf VLMs

TRT is not specific to our depth testbed. We also apply it to three off-the-shelf “visual reasoning” methods — Mirage, Mull-Tokens, and CoVT — each on its own primary benchmark. Each card stands on its own; the within-method delta is the signal.

Three TRT Signatures

When any of these show up in a latent-token method, the gain is unlikely to be driven by token content. Each can produce accuracy gains without the model reading the tokens' content.

Takeaway

Token Replacement Test (TRT) is a task-agnostic, intervention-based diagnostic for latent visual reasoning in vision-language models. By replacing the visual reasoning span while holding prompt, image, token budget, and decoding fixed, TRT exposes whether a model truly uses the content of its visual reasoning tokens. Across our depth testbed and three off-the-shelf methods (Mirage, Mull-Tokens, CoVT), visual reasoning tokens are often not causally used at inference. Accuracy gains alone are not evidence of utilization, and TRT should accompany any claim about latent visual reasoning tokens.

Report TRT alongside accuracy. A simple, inference-time diagnostic. Run it before claiming your model uses its visual reasoning tokens.

Citation

If you find this work useful, please cite:

@article{zhang2026ablate,
  title         = {Ablate-to-Validate: Are Vision-Language Models Really Using Continuous Thought Tokens?},
  author        = {Zhang, Tianyi and Bigverdi, Mahtab and Krishna, Ranjay},
  year          = {2026},
  eprint        = {2605.21642},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}