C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation

This repo contains a proxy-centric alignment framework (C-3PO) that bridges the gap between retrievers and LLMs. Instead of modifying existing components in RAG, C-3PO introduces multi-agent system within a lightweight proxy model to simulate humen-like behaviors that optimizes the entire RAG pipeline while maintaining plug-and-play compatibility.

:boom: News

• [2025.05.01] Our C-3PO is accepted by ICML 2025.
• [2025.03.02] Release our Code.
• [2025.02.12] Release our Demo on the ModelScope.
• [2025.02.10] Release our paper C-3PO on the Arxiv.

:honeybee: Deploy LLM and Retrieval services

Step1: Python Environment

For C-3PO (also works for LLM server)

conda create -n c3po python=3.11
conda activate c3po
pip install -r requirements.txt

For Retrieval (dense model)

conda create -n faiss python=3.11
conda activate faiss
pip install -r retrieval_requirements.txt

Step2: Download LLM from Huggingface

Please download the following models from huggingface:

Qwen2-0.5B
Qwen2-1.5B
Qwen2-72B-Instruct
contriever-msmarco

Step3: Download the wikipedia 2018 dump

Download preprocessed passage data of the wikipedia 2018 dump.

cd ./C-3PO/deploy_servers/retrieve_server/wiki18
wget https://dl.fbaipublicfiles.com/dpr/wikipedia_split/psgs_w100.tsv.gz

Then, download the embedded passages. We use Contriever-MSMARCO.

cd ./C-3PO/deploy_servers/retrieve_server/wikipedia_embeddings
wget https://dl.fbaipublicfiles.com/contriever/embeddings/contriever-msmarco/wikipedia_embeddings.tar

Step4: Deploy the LLM server

cd ./deploy_servers/llm_server
bash qwen_72b_serve.sh

Step5: Deploy the Retrieval server

cd ./C-3PO/deploy_servers/retrieve_server/retrieve_code/wiki18
bash start_wiki18.sh

:dart: Inference

Download our released ckpt (Optional)

You can download our ckpt of C-3PO-1.5B or C-3PO-0.5B on the ModelScope.

Scripts

Our implementation supports two high-performance inference engines: SGLang and vLLM, allowing users to optimize for different deployment scenarios and hardware configurations.

cd ./C-3PO/inference
bash single_model.sh

Tree-structured Rollout for Seed Data (Supervised Warm-up)

We collect seed data through tree-structured rollout using Qwen-2-72B-Instruct.

Step1: tree-structured rollout

cd ./C-3PO/instruct_sampling_scripts
bash run_72b.sh

Step2: supervised fine-tuning

We use Llama-Factory as our training framework for sft.

git clone (from llama-Factory)
# we release our training hyper-parameters for easy reproduction.
cd ./C-3PO/train/sft_scripts
bash run_base_packing.sh

:heart: Acknowledgements

This work is built upon several excellent open-source projects. We sincerely thank:

• Llama Factory for providing the supervised fine-tuning framework
• vLLM for the efficient inference engine with high throughput
• SGLang for the efficient inference engine with high throughput
• OpenRLHF for the comprehensive reinforcement learning framework

We express our gratitude to all these projects for their outstanding contributions to the open-source community.

Citation

If you find our work useful in your research, please consider citing our paper:

@article{chen2025c,
  title={C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation},
  author={Chen, Guoxin and Liao, Minpeng and Yu, Peiying and Wang, Dingmin and Qiao, Zile and Yang, Chao and Zhao, Xin and Fan, Kai},
  journal={arXiv preprint arXiv:2502.06205},
  year={2025}
}

Your support by starring ⭐ this repository would be greatly appreciated!