AI Agents Get Smarter Without Retraining Using New Inference-Time Ranking

Lee Douglas, Deep Tech Correspondent

Researchers are unlocking a new level of adaptability for AI agents navigating complex digital worlds, sidestepping the costly and time-consuming process of retraining. A novel technique called "Best-of-Q" allows powerful Vision-Language Models (VLMs) to improve their decision-making on the fly, using a lightweight Q-function to re-rank candidate actions proposed by a frozen VLM policy. This breakthrough promises to make AI agents more robust in rapidly evolving environments like the internet, where static models quickly become outdated.

Enhancing Agentic Policies Without Retraining

Vision-Language Models have become the workhorses for AI agents tasked with interacting with digital interfaces, from browsing the web to managing operating systems. However, their efficacy is often hampered by the dynamic nature of these environments. Traditional approaches to keeping these models up-to-date involve extensive fine-tuning, a process demanding significant computational resources and large datasets. This is where Best-of-Q offers a compelling alternative.

The core idea is to decouple the VLM's role as an action generator from the final action selection. The VLM, acting as a sophisticated proposer, generates a slate of potential actions given the current state. Subsequently, a separate, offline-trained Q-function—a standard tool in reinforcement learning for estimating action values—steps in. This Q-function doesn't retrain the VLM; instead, it efficiently reranks the VLM's proposed actions. The agent then selects the action deemed most valuable by the Q-function. This approach allows for immediate policy improvement at inference time, a significant departure from methods that rely on retraining the entire model.

The impact of this technique is quantifiable. On the academic WebVoyager benchmark, a Qwen2.5-VL-7B agent, initially achieving a 38.8% success rate, saw its performance jump to 55.7% after employing Best-of-Q. Even a highly capable proprietary GPT-4.1 agent experienced a notable improvement, rising from an 82.4% success rate to 88.8%. These figures underscore the practical benefits of this inference-time optimization, demonstrating its ability to refine agent behavior without the need for costly model updates.

Addressing the Multimodal Data Dilemma

While the focus on VLM agents is a significant step, the broader field of multimodal deep learning is also grappling with challenges in real-world deployment, particularly concerning incomplete data. A separate but related development, dubbed DyMo (Dynamic Modality Selection), tackles the "discarding-imputation dilemma" in multimodal classification. This dilemma arises when dealing with data where some modalities (e.g., text, image, audio) are missing. Existing methods often either discard the incomplete data, risking the loss of crucial information, or attempt to impute the missing parts, potentially introducing noise.

DyMo introduces a framework that adaptively selects and integrates reliable modalities at inference time. It moves beyond the simplistic binary choice of discard or impute by dynamically identifying which modalities, including recovered ones, are most relevant for a given test sample. This selection is guided by a novel algorithm designed to maximize task-relevant information, a proxy for which is derived from the task loss computed during inference.

The research establishes a theoretical link between information content and task loss, allowing for a principled reward function to steer the modality selection process. The framework is architecturally flexible, designed to accommodate arbitrary combinations of modalities, and is supported by a training strategy that promotes robust representation learning. Experiments across diverse natural and medical image datasets have shown DyMo to significantly outperform existing methods in handling various missing-data scenarios.

These two distinct research threads, though addressing different facets of AI agent and model improvement, highlight a shared trend: a move towards more dynamic, efficient, and context-aware AI systems. Best-of-Q optimizes action selection for agents without costly retraining, while DyMo offers a sophisticated way to leverage incomplete multimodal data by making intelligent choices about modality inclusion at inference. Both approaches reflect a maturing understanding of how to deploy AI effectively in unpredictable, real-world conditions, moving beyond brute-force training to more nuanced, adaptive strategies. The ability to enhance performance by intelligently selecting or ranking information at the time of use is becoming a crucial differentiator for practical AI applications, promising more reliable and capable AI agents across a wider array of tasks.