AgentSearch Challenge

The AgentSearch challenge tackles a critical emerging problem in the agentic AI landscape: given a task specification, how do you reliably find and rank the best agent for the job? As the ecosystem of AI agents grows rapidly; spanning diverse providers, capabilities, and specialisations, the ability to discover the right agent at the right time becomes a foundational infrastructure challenge.

This shared task explores agent search in the wild by evaluating retrieval and reranking methods against execution-grounded performance signals across nearly 10,000 real-world agents drawn from multiple providers. Recent studies have shown that semantic similarity alone is insufficient for identifying high-performing agents. Yet it remains an open question which retrieval approaches best capture true agent utility, which ranking signals most reliably predict execution performance, whether execution-aware signals can close the gap between semantic similarity and actual task success, and how well current methods generalise across diverse task types and agent providers. This challenge is designed to systematically investigate these questions.

The challenge consists of two complementary tracks:

• Track 1: Retrieval – Given a natural language task specification, participants develop methods to retrieve the most relevant agents from a large-scale pool. This track evaluates the ability to surface strong candidate agents efficiently and accurately.
• Track 2: Reranking – Given a retrieved set of candidate agents, participants develop methods to rerank them using execution-aware signals, contextual features, and agent metadata. This track focuses on closing the gap between surface-level relevance and true task performance.

Together, the tracks provide the first comprehensive benchmark for agent search, offering the community a shared foundation for developing and comparing principled methods at the intersection of information retrieval and agentic systems.

We release the data for AgentSearchBench on HuggingFace Datasets. You can also access the data from Google Drive.

To download the validation set:

from datasets import load_dataset 
ds = load_dataset("AgentSearch/AgentSearchBench-Tasks", "single-agent_task_query", split="validation")
ds = load_dataset("AgentSearch/AgentSearchBench-Tasks", "multi-agent_task_query", split="validation")
ds = load_dataset("AgentSearch/AgentSearchBench-Tasks", "task_description", split="validation")

To download the test set:

from datasets import load_dataset 
ds = load_dataset("AgentSearch/AgentSearchBench-Tasks", "single-agent_task_query", split="test")
ds = load_dataset("AgentSearch/AgentSearchBench-Tasks", "multi-agent_task_query", split="test")
ds = load_dataset("AgentSearch/AgentSearchBench-Tasks", "task_description", split="test")

Here is a brief description of dataset columns:

• id: Unique identifier for each task.
• task: Task content.
• labels: Binary retrieval labels.
• ranking_labels: Graded ranking labels.
• ref_agents: Reference agents used to generate the task.
• ref_subtasks: Associated subtasks (multi-agent task query and task description).
• rubric: Subtask selection rubric (task description only).

To download the agentbase dataset:

from datasets import load_dataset
ds = load_dataset("AgentSearch/AgentSearchBench-Agents")
df = pd.DataFrame(ds["agents"])

Our benchmarking tasks were generated from execution traces from 2 of the available platforms. You MUST use the same version of the agentbase dataset for the challenge.

active_platforms = ["agentainetwork", "openaiagents"]
 agentbase_executable_df = df[df["platform_name"].isin(active_platforms)]

You will need to register your team when you are submitting your results using the following Google Form. Participants will submit lists of relevant agent ids per query using a standard IR-style format. Teams may submit multiple runs, and reproducibility submissions are encouraged.

Retrieval Track Submission

Binary relevance labels (0 or 1) should be provided for each task in the test set in a JSONL file.

{"q:0": {"agt:openaiagents:55dad6@v1.1": 1}}
{"q:1": {"agt:openaiagents:b950b8@v1.1": 1, "agt:openaiagents:d40350@v1.1": 1, "agt:openaiagents:626021@v1.1": 1}}
{"q:2": {"agt:agentainetwork:2d0fdc@v1.1": 1, "agt:openaiagents:011a48@v1.1": 1}}

Reranking Track Submission

Graded ranking labels should be provided for each task in the test set in a JSONL file. Our reranking track focuses on top 20 agents per task.

{"q:0": {"agt:openaiagents:55dad6@v1.1": 7, "agt:openaiagents:011a48@v1.1": 6, ...}}
{"q:1": {"agt:openaiagents:b950b8@v1.1": 10, "agt:openaiagents:d40350@v1.1": 5, "agt:openaiagents:626021@v1.1": 3, ...}}
{"q:2": {"agt:agentainetwork:2d0fdc@v1.1": 3, "agt:openaiagents:011a48@v1.1": 2}, ...}