AgentSearch

Large language models and AI agents are increasingly deployed as decision-making systems, yet they are still primarily evaluated using static offline benchmarks and costly online experiments. This mismatch limits our ability to measure incremental progress, compare systems reliably, and iterate quickly.

At their core, LLMs and agents are policies: distributions over actions conditioned on context. The sequential decision-making literature has developed decades of theory and algorithms for evaluating such policies, including off-policy evaluation from logged interactions. Recent advances allow us to estimate distributions of treatment effects while navigating fundamental bias-variance tradeoffs.

This talk argues that these ideas provide a natural foundation for the next generation of LLM and agent evaluation. I will discuss how policy evaluation can be extended to the distinctive challenges posed by modern agents-including enormous action spaces, ambiguous rewards, and non-Markovian interactions-and outline opportunities for research at the intersection of sequential decision-making and information retrieval.

Modern information access ecosystems consist of mixtures of systems, such as retrieval systems and large language models, and increasingly rely on marketplaces to mediate access to models, tools, and data, making competition between systems inherent to deployment. In such settings, outcomes are shaped not only by benchmark quality but also by competitive pressure, including user switching, routing decisions, and operational constraints. Yet evaluation is still largely conducted on static benchmarks with accuracy-focused measures that assume systems operate in isolation. This mismatch makes it difficult to predict post-deployment success and obscures competitive effects such as early-adoption advantages and market dominance.

We introduce Marketplace Evaluation, a simulation-based paradigm that evaluates information access systems as participants in a competitive marketplace. By simulating repeated interactions and evolving user and agent preferences, the framework enables longitudinal evaluation and marketplace-level metrics, such as retention and market share, that complement and can extend beyond traditional accuracy-based metrics. We formalize the framework and outline a research agenda, motivated by business and economics, around marketplace simulation, metrics, optimization, and adoption in evaluation campaigns like TREC.

I will examine the emerging problem of Agent Search through the lens of present and past search paradigms. I will begin by introducing the TREC Million LLM Track, which can be viewed as a large-scale evaluation campaign for a form of multi-agent search. I will outline the design of the track, the tasks it poses, and early findings from participating groups, highlighting what these results reveal about the retrieval, ranking, and evaluation of AI systems as search targets in their own right.

I will then contextualise current developments by examining earlier efforts in distributed search and distributed digital libraries, including peer-to-peer systems such as Kazaa and prior research on federated information access. Why did these efforts often produce systems that struggled to outperform a centralised web search such as Google? I will ask when distributed search has in fact succeeded, and what those successes can teach us about the future of agent search.

We are already seeing clues of an emerging information landscape, where publishers and information providers expose content through proprietary LLMs. Current large scale RAG systems synthesise evidence across multiple collections; will we see them start to synthesise across agents as well?

Dense retrieval is trained almost universally with contrastive learning over (query, positive, negative) pairs, a recipe that is expensive to curate and teaches only a coarse, binary notion of relevance. Two benchmarks make the cost concrete: on CoQuIR, state-of-the-art retrievers are largely blind to code quality, ranking buggy, inefficient, insecure, or deprecated code as highly as sound code; on CodeMMR, strong embedders fail on multimodal code retrieval, and each new axis of difficulty demands yet another mountain of curated pairs.

I argue the real bottleneck is the learning signal itself, and present an alternative: supervise retrievers with a language-modeling objective. The core method, Revela, trains a dense retriever from raw, unlabeled text through a single next-token-prediction loss: retriever-computed similarities reweight cross-chunk attention inside a language model, so that predicting one chunk's tokens from its in-batch neighbors backpropagates jointly into both the LM and the retriever, with no queries, labels, or negative mining. Despite using a fraction of the data and compute, Revela matches or surpasses supervised and API retrievers on code, reasoning, and general benchmarks.

The same architecture extends to supervision: ElicitR adds this language-modeling signal as a regularizer alongside contrastive training, eliciting latent reasoning ability that pure contrastive fine-tuning tends to suppress. Together, they show that a language model's own predictive signal is a rich, scalable, and largely untapped source of supervision for retrieval.