AgentBench Explained: Grading LLMs as Multi-Step Agents
AgentBench is a benchmark for evaluating LLMs as agents across a sequence of steps — from receiving an instruction through taking an action, observing the environment's response, and repeating the cycle until it completes the task, fails, or reaches the interaction limit.
Introduced in 2023, THUDM's AgentBench study exposed 29 LLMs to eight environments covering operating systems, databases, knowledge graphs, games, household tasks, web shopping, and web browsing.
In this post, we'll explain how AgentBench works, what its eight environments measure, how the scoring system combines different task types, what the original results revealed, and how the benchmark has evolved with AgentBench FC.
The Episodes
AgentBench was designed as a multi-step evaluation of the reasoning and decision-making abilities of LLM-based agents during interactions within complex environments.
Each task gives the model a defined way to interface with the environment. An operating-system agent can issue Bash commands, a database agent issues SQL queries, and a web-shopping agent has to navigate an online store toward a specified goal. Every action produces new information that can affect the model's next move.
Every AgentBench episode follows this basic structure:
instruction → action → environment response → next action → task result
That sequence resembles a multi-step agent workflow: the model has to preserve the goal, interpret observations correctly, use the available interface, and decide what to do next as the task develops.
A model can understand the initial instruction and make a sensible first move yet fail later because it misreads an observation, issues an invalid action, loses track of an earlier result, or stops before completing the task.
The Environments
The original AgentBench suite contains eight environments grouped into code-grounded, game-grounded, and web-grounded tasks. Five were created for AgentBench, while House-Holding, Web Shopping, and Web Browsing adapt existing evaluation environments.
| Environment | What the agent has to do | Main metric |
|---|---|---|
| Operating System (OS) | Use Bash commands in an Ubuntu environment to answer questions or complete system tasks | Success rate |
| Database (DB) | Operate on real databases through SQL across multiple tables and query types | Success rate |
| Knowledge Graph (KG) | Navigate a partially observable knowledge graph to answer questions | Answer F1 |
| Digital Card Game (DCG) | Play a turn-based card game against another agent | Reward |
| Lateral Thinking Puzzles (LTP) | Ask questions and reconstruct the hidden events behind a situation puzzle | Game progress |
| House-Holding (HH) | Complete household tasks in the ALFWorld text environment | Success rate |
| Web Shopping (WS) | Search and navigate a simulated online store to select a product matching the instruction | Reward |
| Web Browsing (WB) | Perform browser actions such as clicking, selecting, and typing across Mind2Web tasks | Step success rate |
The environments vary in both interaction length and success criteria. Operating-system tasks test executable command sequences, lateral-thinking puzzles depend on information accumulated through repeated exchanges, and the knowledge-graph environment gives the agent incomplete information and requires it to decide which entities or relations to inspect next.
AgentBench separates the environment from the model interface through three components:
- Task Server hosts the task and returns observations
- Agent Server generates actions from the interaction history
- Client coordinates the exchange between them.
This setup lets different models interact with the same task infrastructure.
The Scoring
AgentBench uses dedicated, not-directly-comparable metrics for different environments: Operating System, Database, and House-Holding use success rate; Knowledge Graph uses answer F1; Digital Card Game and Web Shopping use reward; Lateral Thinking Puzzles use game progress; and Web Browsing uses step success rate.
For the overall AgentBench score, the researchers calculated the average performance of the evaluated models on each task and used the reciprocal of that average as a fixed task weight. Multiplying each model's task score by the corresponding weight rescales the environments so their average contribution is comparable before the eight results are combined.
The overall score condenses eight different metrics into one aggregate number for ranking models side by side; it doesn't correspond to a percentage of tasks completed.
Per-environment results can tell us more when the intended agent has a specific job. Models can have similar overall AgentBench scores while performing very differently on SQL interaction, knowledge-graph queries, shell commands, or web navigation.
The Original AgentBench Results
The published ICLR 2024 evaluation tested 29 API-based and open-source LLMs.
GPT-4 achieved the highest overall AgentBench score at 4.01 (since each task's average model performance is rescaled to a baseline of 1, it performed roughly 4× better than the field average across the eight environments combined).
The researchers reported a substantial performance disparity between the strongest commercial models and many of the open-source models evaluated at the time. Performance also varied across environments, which made the aggregate ranking only part of the result story.
The paper's failure analysis identified poor long-term reasoning, decision-making, and instruction following as recurring obstacles. Agent failures could develop across several interactions as an earlier mistake, malformed action, or misunderstood observation affected later decisions.
While the study scores capture older model performance (2023–2024), the failure analysis still has relevance because it showed how multi-step interaction reveals coordination breakdowns that a single-response evaluation would never catch.
What Changed With AgentBench FC
On October 10, 2025, THUDM introduced AgentBench FC, a function-calling version integrated with AgentRL, its multitask and multi-turn framework which allowed the same containerized task environments used to score a model to now also train one through reinforcement learning.
The current AgentBench repository uses a function-calling prompt format and provides fully containerized deployment for five task families:
- ALFWorld
- Database / DBBench
- Knowledge Graph
- Operating System
- WebShop
The original v0.1 and v0.2 implementations are still available in the repository, while the current version uses AgentBench FC to update the interface and deployment for a subset of the original benchmark.
THUDM also introduced VisualAgentBench in 2024, which extends the research into multimodal agents through five environments covering embodied tasks, graphical interfaces, and visual design.
Where AgentBench Ends and Memory Evaluation Begins
An AgentBench score is assigned as the outcome of the whole interactive task. Planning, instruction following, tool interaction, action formatting, interpretation of environment feedback, and information carried across earlier steps can all influence whether the agent completes the task.
Because these capabilities contribute to the same result, AgentBench doesn't isolate memory quality on its own. The paper's finding that models struggled with "long-term reasoning" reflects performance across an ongoing interaction rather than persistent memory across separate tasks or sessions.
AI memory benchmarks evaluate that narrower capability more directly, including whether earlier information can be retrieved after long histories, updated when newer evidence arrives, and returned during later interactions.
When information learned during one task needs to influence another, the agent needs memory that persists beyond the current interaction. cognee provides long-term memory to agents that combines graph, vector, and lexical retrieval so earlier knowledge can be stored with context and retrieved when later work depends on it.
FAQ
Answers to the most common questions from this guide.
Who's behind AgentBench?
AgentBench was developed by researchers from Tsinghua University and collaborating institutions. The original paper, AgentBench: Evaluating LLMs as Agents, was released in 2023 and later published at ICLR 2024. The official implementation is maintained under THUDM.
Is AgentBench open source?
Yes. The AgentBench code and task infrastructure are publicly available through THUDM's GitHub repository. The repository also contains the newer AgentBench FC implementation, while earlier versions of the original benchmark remain available through its version history.
Is AgentBench the same as VisualAgentBench?
No. AgentBench evaluates language-model agents across text-based and interactive environments such as operating systems, databases, knowledge graphs, and web tasks.
VisualAgentBench is a related THUDM project introduced later for multimodal agents. Its environments involve visual perception and interaction across embodied tasks, graphical interfaces, and visual-design workflows.
Is AgentBench still relevant?
Yes. AgentBench still provides a clear framework for evaluating how LLM agents handle multi-step tasks across different environments, although its original leaderboard reflects an earlier generation of models.
AgentBench FC extends five task families to function-calling agents — closer to how most production agents actually call tools today — giving the project a more current implementation. For model selection today, its results are most informative alongside newer benchmarks focused on the specific capability or environment being tested.
