github.com/AgileRL/AgileRL @v2.7.1

Reinforcement learning streamlined.

Easier and faster reinforcement learning with RLOps. Visit our website. View documentation.

Join the Discord Server for questions, help and collaboration.

🚀 Train super-fast for free on Arena, the RLOps platform from AgileRL 🚀

AgileRL is a Deep Reinforcement Learning library focused on improving development by introducing RLOps - MLOps for reinforcement learning.

This library is initially focused on reducing the time taken for training models and hyperparameter optimization (HPO) by pioneering evolutionary HPO techniques for reinforcement learning.

Evolutionary HPO has been shown to drastically reduce overall training times by automatically converging on optimal hyperparameters, without requiring numerous training runs.

We are constantly adding more algorithms and features. AgileRL already includes state-of-the-art evolvable on-policy, off-policy, offline, multi-agent and contextual multi-armed bandit reinforcement learning algorithms with distributed training.

AgileRL offers 10x faster hyperparameter optimization than SOTA.

Table of Contents

• Get Started
• Benchmarks
• Tutorials
• Algorithms implemented
• Train an agent
• Citing AgileRL

Get Started

To see the full AgileRL documentation, including tutorials, visit our documentation site. To ask questions and get help, collaborate, or discuss anything related to reinforcement learning, join the AgileRL Discord Server.

Install as a package with pip:

pip install agilerl

Or install in development mode:

git clone https://github.com/AgileRL/AgileRL.git && cd AgileRL
pip install -e .

If you wish to install all additional dependencies please specify [all] or if you want to install a specific family of dependencies specify that family directly. At present, we have just one family, [llm], which contains the dependencies related to our LLM RFT algorithms (datasets, deepspeed, peft, transformers, vllm).

pip install agilerl[all]

Or in development mode:

pip install -e ".[all]"

To install the nightly version of AgileRL with the latest features, use:

pip install git+https://github.com/AgileRL/AgileRL.git@nightly

Benchmarks

Reinforcement learning algorithms and libraries are usually benchmarked once the optimal hyperparameters for training are known, but it often takes hundreds or thousands of experiments to discover these. This is unrealistic and does not reflect the true, total time taken for training. What if we could remove the need to conduct all these prior experiments?

In the charts below, a single AgileRL run, which automatically tunes hyperparameters, is benchmarked against Optuna's multiple training runs traditionally required for hyperparameter optimization, demonstrating the real time savings possible. Global steps is the sum of every step taken by any agent in the environment, including across an entire population.

AgileRL offers an order of magnitude speed up in hyperparameter optimization vs popular reinforcement learning training frameworks combined with Optuna. Remove the need for multiple training runs and save yourself hours.

AgileRL also supports multi-agent reinforcement learning using the Petting Zoo-style (parallel API). The charts below highlight the performance of our MADDPG and MATD3 algorithms with evolutionary hyper-parameter optimisation (HPO), benchmarked against epymarl's MADDPG algorithm with grid-search HPO for the simple speaker listener and simple spread environments.

Tutorials

We are constantly updating our tutorials to showcase the latest features of AgileRL and how users can leverage our evolutionary HPO to achieve 10x faster hyperparameter optimization. Please see the available tutorials below.

Tutorial Type	Description	Tutorials
Single-agent tasks	Guides for training both on and off-policy agents to beat a variety of Gymnasium environments.	PPO - Acrobot

TD3 - Lunar Lander

Rainbow DQN - CartPole

Recurrent PPO - Masked Pendulum | | Multi-agent tasks | Use of PettingZoo environments such as training DQN to play Connect Four with curriculum learning and self-play, and for multi-agent tasks in MPE environments. | DQN - Connect Four

MADDPG - Space Invaders

MATD3 - Speaker Listener | | Hierarchical curriculum learning | Shows how to teach agents Skills and combine them to achieve an end goal. | PPO - Lunar Lander | | Contextual multi-arm bandits | Learn to make the correct decision in environments that only have one timestep. | NeuralUCB - Iris Dataset

NeuralTS - PenDigits | | Custom Modules & Networks | Learn how to create custom evolvable modules and networks for RL algorithms. | Dueling Distributional Q Network

EvolvableSimBa | | LLM Finetuning | Learn how to finetune an LLM using AgileRL. | GRPO

GRPO with HPO

SFT & DPO

Multi-turn LLMPPO, LLMREINFORCE & GRPO |

Evolvable algorithms (more coming soon!)

### Single-agent algorithms

RL	Algorithm
On-Policy	Proximal Policy Optimization (PPO)
Off-Policy	Deep Q Learning (DQN)

Rainbow DQN

Deep Deterministic Policy Gradient (DDPG)

Twin Delayed Deep Deterministic Policy Gradient (TD3) | | Offline | Conservative Q-Learning (CQL)

Implicit Language Q-Learning (ILQL) |

### Multi-agent algorithms

RL	Algorithm
Multi-agent	Multi-Agent Deep Deterministic Policy Gradient (MADDPG)

Multi-Agent Twin-Delayed Deep Deterministic Policy Gradient (MATD3)

Independent Proximal Policy Optimization (IPPO)|

### Contextual multi-armed bandit algorithms

RL	Algorithm
Bandits	Neural Contextual Bandits with UCB-based Exploration (NeuralUCB)

Neural Contextual Bandits with Thompson Sampling (NeuralTS) |

### LLM Fine-tuning Algorithms

Type	Algorithm
On-Policy RL	Group Relative Policy Optimization (GRPO)

Clipped IS-weight Policy Optimization (CISPO)

Group Sequence Policy Optimization (GSPO)

LLM Proximal Policy Optimization (LLM PPO)

LLM REINFORCE | | Preference Optimization | Direct Preference Optimization (DPO) | | Supervised Fine-Tuning | Supervised Fine-Tuning (SFT) |

Train an Agent to Beat a Gym Environment

Before starting training, there are some meta-hyperparameters and settings that must be set. These are defined in INIT_HP, for general parameters, and MUTATION_PARAMS, which define the evolutionary probabilities, and NET_CONFIG, which defines the network architecture. For example:

Basic Hyperparameters

```python INIT_HP = { 'ENV_NAME': 'LunarLander-v3', # Gym environment name 'ALGO': 'DQN', # Algorithm 'DOUBLE': True, # Use double Q-learning 'CHANNELS_LAST': False, # Swap image channels dimension from last to first [H, W, C] -> [C, H, W] 'BATCH_SIZE': 256, # Batch size 'LR': 1e-3, # Learning rate 'MAX_STEPS': 1_000_000, # Max no. steps 'TARGET_SCORE': 200., # Early training stop at avg score of last 100 episodes 'GAMMA': 0.99, # Discount factor 'MEMORY_SIZE': 10000, # Max memory buffer size 'LEARN_STEP': 1, # Learning frequency 'TAU': 1e-3, # For soft update of target parameters 'TOURN_SIZE': 2, # Tournament size 'ELITISM': True, # Elitism in tournament selection 'POP_SIZE': 6, # Population size 'EVO_STEPS': 10_000,