Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning

Reasoning agents increasingly rely on external tools such as web search to answer complex queries. Reinforcement learning (RL) finetuning algorithms such as GRPO have improved long-form reasoning in text-only language models, particularly for coding and mathematics. Reliable tool use in multimodal agents, however, remains challenging because models must interpret text and images while integrating noisy retrieved evidence, often under sparse outcome-level supervision without explicit verification signals. We present Self-Verification via Reinforcement Learning (SVRL), an RL-only finetuning framework that trains multimodal agents to verify and filter retrieved evidence within their own reasoning traces, reducing reliance on external verifiers at inference time. SVRL also introduces a search-aware penalty that discourages unnecessary tool calls and a query-diversity reward that encourages diverse, well-formed search queries, providing fine-grained feedback on when and what to search. Finetuning Qwen-2.5-VL-7B with SVRL on only 5,000 visual question answering examples yields consistent gains in multi-hop VQA generalization and tool efficiency across benchmarks. Overall, SVRL narrows the gap between compact agents and much larger proprietary models while requiring substantially lower training and inference cost.

Step through what the agent actually did: its reasoning, when it called a tool, what came back, which evidence it kept, and the answer it committed to. Switch models to see how the baseline handled the same question.

Analyzing a GRPO-finetuned search agent on FVQA surfaces three recurring failure modes. First, agents are poorly calibrated about when to search: search was required but not invoked in over 10% of cases, while over 30% triggered unnecessary search. Second, tool outputs are noisy and go unfiltered. Nearly 45% of retrieved results were irrelevant, and the agent still answered incorrectly in over 28% of cases even when it issued an appropriate query and relevant evidence was present. Third, outcome-only supervision leaves every intermediate decision unlabeled, so distinct failure causes collapse into the same terminal penalty.

SVRL treats the multimodal agent as a stochastic policy over multi-turn trajectories and finetunes it with a GRPO-style objective, replacing the sparse trajectory reward with a product of fine-grained factors:

• Search-aware calibration (raware): a single tool-free forward pass labels each training instance as search_free or not, and rollouts that call a tool on a search_free instance are downweighted. This targets avoidable searches without suppressing necessary ones.
• Query diversity (rcount): the agent proposes k candidate queries and executes one; the fraction of unique, well-formed proposals scales the task reward.
• Verification alignment (rqalign, rsalign): the agent emits a binary usefulness vector over retrieved items (e.g. <verify> 0,1,0,1 </verify>) plus a short justification. A train-time oracle verifier with access to the ground-truth answer scores the same queries and snippets, and the agent is rewarded for agreeing with it.

The verification factors are gated on final-answer correctness to avoid reward hacking, and the oracle verifier is discarded at inference. The deployed agent verifies on its own.

We train Qwen-2.5-VL-7B-Instruct on 5,000 FVQA examples and evaluate in-distribution (FVQA-test, InfoSeek) and out-of-distribution (MMSearch, LiveVQA, SimpleVQA). Acc is answer accuracy; SR is search ratio, the fraction of questions on which the agent chose to search.

Training dynamics

Test-time scaling

On a 256-example subset of FVQA-test we allow up to K parallel text searches per question. SVRL already proposes up to five candidate queries within one reasoning step, so increasing K first uses those directly and then samples additional proposals at gradually higher temperature. Candidate answers are aggregated by exact majority, falling back to a nearest-neighbour consensus over answer embeddings.

Search calibration

Splitting FVQA by whether the agent chose to search shows the gain is not just from searching more. SVRL searches considerably less and is more accurate on both subsets.

At inference the model generates an average of 4.86 unique queries per text-search step, so the policy does not collapse to a single repeated query template despite the fixed five-query interface.

Where the errors go

Conditioning on cases where an external judge rates the issued query as good, SVRL still improves, so the gains extend past query generation into downstream evidence use.

How good is the learned verifier?

Comparing the agent's own <verify> mask against oracle filtering labels over all retrieved summaries, agreement is 71.3%. The disagreements are strongly asymmetric: the learned filter is conservative, rarely discarding useful evidence but frequently keeping noise.

A second backbone, and proprietary baselines

The same recipe applied to Qwen3-VL-8B performs comparably to SVRL-7B, so the gains are not tied to one backbone. Both SVRL variants also score above a much larger proprietary system in the search-based regime.

The agent is prompted in a ReAct style: every tool call must be preceded by a <reason> block, and after any tool call the next turn must emit a binary verification mask before the following action. Each episode allows at most three interaction steps (one image search and up to two text searches) within an 8124-token budget.

System message.
You are a helpful assistant who will be evaluated on a Visual Question Answering
Task. You should strictly follow a reason-to-act process to answer a user-provided
question about an image. All thinking must be written inside <reason> and
</reason> tags.

Available actions.
First analyze the question and any available observations, including the
user-provided image and any retrieved search results. Then choose one of the
following actions: <search><img></search> to perform image search;
<text_search>query1, query2, query3, query4, query5</text_search> to generate
five unique relevant text queries, one of which is selected at random during
training; or <answer>xxxxx</answer> to return the final answer. If an action
was performed previously, output a binary relevance list containing as many comma
separated values as the number of tool output snippets like this:
<verify>0/1, 0/1, ...</verify> before choosing the next action. Retrieved image
and text results are placed inside <information> and </information> tags.

Allowed output formats.
The model output must follow exactly one of the following formats:

  <reason> YOUR THINKING PROCESS </reason>
  <search><img></search>

or

  <reason> YOUR THINKING PROCESS </reason>
  <verify>0/1, 0/1, ...</verify>
  <text_search> FIVE GENERATED COMMA SEPARATED UNIQUE TEXT QUERIES </text_search>

or

  <reason> YOUR THINKING PROCESS TO ASSESS TOOL OUTPUTS AND ANSWER THE QUESTION </reason>
  <verify>0/1, 0/1, ...</verify>
  <answer> YOUR ANSWER AFTER GETTING ENOUGH INFORMATION </answer>

Additional instructions.
If an image or text search call was made previously, the model should first reason
about why each returned summary was helpful or unhelpful and then output a binary
relevance list inside <verify> and </verify> before producing the final answer.
If no prior search call was made, output an empty <verify> </verify> block. The
number of binary scores must match the number of retrieved summaries. The final
answer must appear only inside <answer> and </answer> tags, without extra
explanation. If the question expects a year, date, quantity or location, be as
specific as possible and mention appropriate units. If the question is yes-or-no,
answer only yes or no.

@misc{sathish2026svrl,
  title         = {Eliciting Self-Verification in Multimodal Reasoning Agents
                   with Reinforcement Learning},
  author        = {Sathish, Vishwas and Ranjan, Viresh and Zhu, Xinliang
                   and Dhua, Arnab and Gray, Douglas},
  year          = {2026},
  eprint        = {2609.08025},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2609.08025}
}