C^3-Bench: The Things Real Disturbing LLM based Agent in Multi-Tasking

📖 <a>English</a> •
<a href="https://github.com/Tencent-Hunyuan/C3-Benchmark/raw/main/README_ZH.md">中文</a>



🤗 <a href="https://huggingface.co/datasets/tencent/C3-BenchMark">Dataset</a> •
📚 <a href="https://arxiv.org/abs/2505.18746">Preprint Paper</a>

🎆 News

• This repo has moved to https://github.com/yupeijei1997/WildToolBench, Please check our latest progress there.

• 2026.1.26 🎉🎉🎉 Our paper Benchmarking LLM Tool-Use in the Wild is accepted by ICLR 2026!

📖 Overview

Agents based on large language models leverage tools to modify environments, revolutionizing how AI interacts with the physical world. Unlike traditional NLP tasks that rely solely on historical dialogue for responses, these agents must consider more complex factors, such as inter-tool relationships, environmental feedback and previous decisions, when making choices. Current research typically evaluates agents via multi-turn dialogues. However, it overlooks the influence of these critical factors on agent behavior. To bridge this gap, we present an open-source and high-quality benchmark C^3-Bench. This benchmark integrates attack concepts and applies univariate analysis to pinpoint key elements affecting agent robustness. In concrete, we design three challenges: navigate complex tool relationships, handle critical hidden information and manage dynamic decision paths. Complementing these challenges, we introduce fine-grained metrics, innovative data collection algorithms and reproducible evaluation methods. Extensive experiments are conducted on 49 mainstream agents, encompassing general fast-thinking, slow-thinking and domain-specific models. We observe that agents have significant shortcomings in handling tool dependencies, long context information dependencies and frequent policy-type switching. In essence, C^3-Bench aims to expose model vulnerabilities through these challenges and drive research into the interpretability of agent performance.

😊 Key Materials

• Test data location: c3_bench/data/C3-Bench.jsonl or 🤗 Dataset
• More detailed information about the C3-Bench can be found below

⚡️ Quickstart

Basic Installation

# Create a new Conda environment with Python 3.10
conda create -n C3-Bench python=3.10
conda activate C3-Bench

# Clone the C3-Bench repository
git clone https://github.com/Tencent-Hunyuan/C3-Benchmark.git

# Change directory to the `c3_bench`
cd c3_bench/

# Install the package
pip install -r requirements.txt

⏳ Inference

💾 Test Data

Address: c3_bench/data/C3-Bench.jsonl

Description: Our test data has undergone five rounds of manual inspection and correction by five senior algorithm researcher with years of experience in NLP, CV, and LLM, taking about one month in total. It boasts extremely high quality and accuracy, with a tight connection between multiple rounds of tasks, increasing difficulty, no unusable invalid data, and complete consistency with human distribution. Its evaluation results and conclusions are of great reference value for subsequent optimization in the Agent direction.

Specifically, the data quality optimization work went through the following stages:

1. The initial data was generated using our proposed Multi Agent Data Generation framework, covering all possible action spaces.

2. The test data was then divided according to four different types of actions defined by us and manually inspected and corrected by four different algorithm researcher. Specifically, since tasks generated by LLM are always too formal and not colloquial enough, especially after the second task, it is difficult to generate true multi-turn tasks. Therefore, we conducted the first round of corrections based on the criteria of colloquialism and true multi-turn tasks. Notably, in designing the third and fourth round tasks, we added tasks with long-term memory, a true multi-turn type, to increase the difficulty of the test set.

Note: In the actual construction process, the four algorithm researcher adopted a layer-by-layer approach, first generating a layer of data with the model, then manually inspecting and correcting it, before generating and correcting the next layer of data. This approach avoids the difficulty of ensuring overall correctness and maintaining data coherence when, after generating all layers of data at once, a problem in one layer requires corrections that often affect both the previous and subsequent layers. Thus, our layer-by-layer construction ensures strong logical consistency and close relationships between layers, without any unreasonable trajectories.

1. After the first round of corrections by the four algorithm researcher, one senior experts in the Agent field would comment on each piece of data, indicating whether it meets the requirements and what problems exist, followed by a second correction by the four algorithm researcher.

2. After the second round of corrections, we introduced cross-validation, where the four algorithm researcher inspected and commented on each other's data. Then, the four algorithm researcher and one senior experts in the Agent field discussed and made a third round of corrections on the doubtful data.

3. After the third round of corrections, the one senior experts in the Agent field separately conducted a fourth round of inspection and correction on all data to ensure absolute accuracy.

4. Finally, since human corrections might introduce errors, we used code to check for possible parameter type errors and unreasonable dependencies caused by manual operations, with one senior experts making the final fifth round of corrections.

Through these five stages of data quality optimization, each piece of data was manually corrected and constructed by multiple algorithm experts, improving our test data's accuracy from less than 60% initially to 100% correctness. The combination of model generation and multiple human corrections also endowed our data with excellent diversity and quality.

At the same time, compared to other benchmarks such as BFCL, T-EVAL, etc., our test data covers all possible action spaces, and in the second to fourth rounds of true multi-turn tasks, the coverage rate has reached two 100%, which also makes our data distribution very balanced, capable of testing out the weaknesses of the model without any blind spots.

Ultimately, this high-quality data set we constructed laid the foundation for our subsequent experiments, lending absolute credibility to our conclusions.

Additionally, we provide bilingual support for the test data, including both English and Chinese versions, all of which have undergone the aforementioned manual inspection process. Subsequent LeadBoard results will primarily report the English version.

🛠️ Framework

Our evaluation framework is constructed in a manner that separates inference and results analysis, offering several advantages as follows:

• High reproducibility: In our test data, the execution results of all tools corresponding to the golden answers have been persistently saved. There is no need for any website's KEY, and there are no unstable tool invocation scenarios, ensuring the reproducibility of the results.
• High evaluation efficiency: Our evaluation is conducted dynamically. The first phase is carried out using EvalByToolCallGraph module, deciding whether to continue calling based on whether the action (predicted tool name) matches the golden answer. Meanwhile, decision tree pruning is used during the process, significantly reducing the number of maintenance paths and speeding up the evaluation.
• High code reusability: All our requests use the standard ToolCalls protocol, making our evaluation code highly reusable. Additionally, we have encapsulated the ToolCalls protocol for several open-source general models and open-source specialized models that did not support the ToolCalls protocol, making the code logic clearer and solving the problem of other evaluation frameworks mixing Prompt and ToolCalls invocation methods, leading to confused logic.
• Multiple evaluation analysis dimensions: After obtaining the prediction and action-level evaluation results from the first phase, we use the AnalysisResult module to conduct a detailed evaluation of its results, including analyses across six dimensions. To our knowledge, among all Agent evaluation frameworks, we offer the most analysis dimensions and the most detailed results. Also, our results are saved in CSV files, facilitating developers in bad case analysis.
• Strong scalability: Since we use the standard ToolCalls protocol, for API models, APIHandle can be used for rapid integration; for new open-source models, we will continue to update this repository for integration; for developers' own trained models, they can refer to our Handle code to encapsulate the Prompt calling method into the ToolCalls protocol for rapid integration and verification.

The overall framework diagram is as follows:

🤖 API Models

This project supports the API model. Taking hunyuan-turbos-latest as an example, set the following key in the environment variable

export MODEL=hunyuan-turbos-latest
export API_KEY=xxxxxxxxx
export BASE_URL=https://api.hunyuan.cloud.tencent.com/v1

After that, use the following code to request the model results and set the model to hunyuan-turbos-latest. If the test stops unexpectedly in the middle, you can modify continue_file to continue the test, which will prevent the predicted results from being predicted repeatedly.

upta format

cd c3_bench/bench_test

python3 request_pipeline_upta.py \
    --model=hunyuan-turbos-latest \
    --data_path=./data/C3-Bench.jsonl \
    --output_path=./result \
    --language=en \
    --continue_file=empty.jsonl \
    --remove_role=True \
    --contain_context=True

ua format

cd c3_bench/bench_test

python3 request_pipeline.py \
    --model=hunyuan-turbos-latest \
    --data_path=./data/C3-Bench.jsonl \
    --output_path=./result \
    --language=en \
    --continue_file=empty.jsonl \
    --remove_role=True \
    --contain_context=True

🤗 HuggingFace Models

This project also supports a variety of open source special models and open source general models. You can choose to use vllm or native Huggingface to deploy the model.

Because vllm supports Tool Calling service deployment and adapts the best System Prompt for Function Call for each model, we recommend starting the service in this way to better evaluate the model capabilities. When the model to be evaluated is not supported in vllm, you can also use the native Huggingface method to deploy the model.

Take using vllm to deploy HunYuan-A13B as an example:

To start the vllm service of Tool Calling, add the following fields to the startup command:

--enable-auto-tool-choice \
--tool-parser-plugin /path/tool_parser/hunyuan_tool_parser.py \
--tool-call-parser hunyuan \

The following is an example of a complete startup script:

MODEL_PATH=${MODEL_PATH}

# export VLLM_LOGGING_LEVEL=DEBUG
export VLLM_HOST_IP=$LOCAL_IP

python3 -m vllm.entrypoints.openai.api_server \
    --host ${LOCAL_IP} \
    --port 8020 \
    --trust-remote-code \
    --model ${MODEL_PATH} \
    --gpu_memory_utilization 0.92 \
    --tensor-parallel-size 2 \
    --dtype bfloat16 \
    --disable-log-stats \
    --enable-auto-tool-choice \
    --tool-parser-plugin /path/tool_parser/hunyuan_tool_parser.py \
    --tool-call-parser hunyuan \
    2>&1 | tee log_server.txt

After the model is deployed, use the following code to request the model result. First, set the environment variable MODEL=Hunyuan-A13B-Instruct, then set model to hunyuan-a13b (to obtain the corresponding handle, view in c3_bench/bench_test/handle/handles.py), and set model_url to the IP and port number of your deployment machine, for example: http://111.111.111.111:12345

If the test stops unexpectedly, you can modify continue_file to continue the test.

upta格式

export MODEL=Hunyuan-A13B-Instruct

python3 request_pipeline_upta.py \
    --model=hunyuan-a13b \
    --data_path=./data/C3-Bench.jsonl \
    --output_path=./result \
    --language=en \
    --model_url=MODEL_URL \
    --continue_file=empty.jsonl \
    --remove_role=True \
    --contain_context=True

ua格式

export MODEL=Hunyuan-A13B-Instruct

python3 request_pipeline.py \
    --model=hunyuan-a13b \
    --data_path=./data/C3-Bench.jsonl \
    --output_path=./result \
    --language=en \
    --model_url=MODEL_URL \
    --continue_file=empty.jsonl \
    --remove_role=True \
    --contain_context=True

In addition, if you want to use native Huggingface for deployment, refer to the following process:

First, you need to download the model to a certain address, and then add the model name and the address to the tool_model_path_map variable in c3_bench/tool_calls/tool_model_map.py.

After that, you can use the following code to deploy the model.

python3 web_server.py MODEL_NAME

💫 Evaluation

Use the following code to evaluate the model prediction results. Fill PREDICT_DATA_FILE with the corresponding prediction file in the ./result directory of the previous step. The evaluation results include: matrix accuracy of action type and layer, accuracy of action type and layer, analysis of multi-tool call results, error type analysis, true and false multi-round accuracy, true multi-round type accuracy, and parameter error type analysis.

The detailed results will be output to data_with_details.csv.

```bash cd c3_bench/bench_test

python3 analysis_result.py \ --data_file PREDICT_DATA_FILE \