comment: <> (

)

comment: <> (

)

MARLlib: A Scalable and Efficient Multi-agent Reinforcement Learning Library

News: We are excited to announce that a major update has just been released. For detailed version information, please refer to the version info.

Multi-agent Reinforcement Learning Library (MARLlib) is a MARL library that utilizes Ray and one of its toolkits RLlib. It offers a comprehensive platform for developing, training, and testing MARL algorithms across various tasks and environments.

Here's an example of how MARLlib can be used:

from marllib import marl

# prepare env
env = marl.make_env(environment_name="mpe", map_name="simple_spread")

# initialize algorithm with appointed hyper-parameters
mappo = marl.algos.mappo(hyperparam_source='mpe')

# build agent model based on env + algorithms + user preference
model = marl.build_model(env, mappo, {"core_arch": "gru", "encode_layer": "128-256"})

# start training
mappo.fit(env, model, stop={'timesteps_total': 1000000}, share_policy='group')

# ready to control
mappo.render(env, model, share_policy='group', restore_path='path_to_checkpoint')

Why MARLlib?

Here we provide a table for the comparison of MARLlib and existing work.

Library	Supported Env	Algorithm	Parameter Sharing	Model
PyMARL	1 cooperative	5	share	GRU
PyMARL2	2 cooperative	11	share	MLP + GRU
MAPPO Benchmark	4 cooperative	1	share + separate	MLP + GRU
MAlib	4 self-play	10	share + group + separate	MLP + LSTM
EPyMARL	4 cooperative	9	share + separate	GRU
MARLlib	12 no task mode restriction	18	share + group + separate + customizable	MLP + CNN + GRU + LSTM

Library	Github Stars	Documentation	Issues Open	Activity	Last Update
PyMARL		:x:
PyMARL2		:x:
MAPPO Benchmark		:x:
MAlib
EPyMARL		:x:
MARLlib

comment: <> (

)

comment: <> (

)

key features

:beginner: MARLlib offers several key features that make it stand out:

• MARLlib unifies diverse algorithm pipelines with agent-level distributed dataflow, allowing researchers to develop, test, and evaluate MARL algorithms across different tasks and environments.
• MARLlib supports all task modes, including cooperative, collaborative, competitive, and mixed. This makes it easier for researchers to train and evaluate MARL algorithms across a wide range of tasks.
• MARLlib provides a new interface that follows the structure of Gym, making it easier for researchers to work with multi-agent environments.
• MARLlib provides flexible and customizable parameter-sharing strategies, allowing researchers to optimize their algorithms for different tasks and environments.

:rocket: Using MARLlib, you can take advantage of various benefits, such as:

• Zero knowledge of MARL: MARLlib provides 18 pre-built algorithms with an intuitive API, allowing researchers to start experimenting with MARL without prior knowledge of the field.
• Support for all task modes: MARLlib supports almost all multi-agent environments, making it easier for researchers to experiment with different task modes.
• Customizable model architecture: Researchers can choose their preferred model architecture from the model zoo, or build their own.
• Customizable policy sharing: MARLlib provides grouping options for policy sharing, or researchers can create their own.
• Access to over a thousand released experiments: Researchers can access over a thousand released experiments to see how other researchers have used MARLlib.

Installation

Note: Currently MARLlib supports Linux only.

Step-by-step (recommended)

• install dependencies
• install environments
• install patches

1. install dependencies (basic)

First, install MARLlib dependencies to guarantee basic usage. following this guide, finally install patches for RLlib.

$ conda create -n marllib python=3.8 # or 3.9
$ conda activate marllib
$ git clone https://github.com/Replicable-MARL/MARLlib.git && cd MARLlib
$ pip install -r requirements.txt

2. install environments (optional)

Please follow this guide.

3. install patches (basic)

Fix bugs of RLlib using patches by running the following command:

$ cd /Path/To/MARLlib/marl/patch
$ python add_patch.py -y

PyPI

$ pip install --upgrade pip
$ pip install marllib

Getting started

Prepare the configuration

There are four parts of configurations that take charge of the whole training process.

• scenario: specify the environment/task settings
• algorithm: choose the hyperparameters of the algorithm
• model: customize the model architecture
• ray/rllib: change the basic training settings

Before training, ensure all the parameters are set correctly, especially those you don't want to change.

Note: You can also modify all the pre-set parameters via MARLLib API.*

Register the environment

Ensure all the dependencies are installed for the environment you are running with. Otherwise, please refer to MARLlib documentation.

task mode	api example
cooperative	marl.make_env(environment_name="mpe", map_name="simple_spread", force_coop=True)
collaborative	marl.make_env(environment_name="mpe", map_name="simple_spread")
competitive	marl.make_env(environment_name="mpe", map_name="simple_adversary")
mixed	marl.make_env(environment_name="mpe", map_name="simple_crypto")

Most of the popular environments in MARL research are supported by MARLlib:

Env Name	Learning Mode	Observability	Action Space	Observations
LBF	cooperative + collaborative	Both	Discrete	1D
RWARE	cooperative	Partial	Discrete	1D
MPE	cooperative + collaborative + mixed	Both	Both	1D
SMAC	cooperative	Partial	Discrete	1D
MetaDrive	collaborative	Partial	Continuous	1D
MAgent	collaborative + mixed	Partial	Discrete	2D
Pommerman	collaborative + competitive + mixed	Both	Discrete	2D
MAMuJoCo	cooperative	Full	Continuous	1D
GRF	collaborative + mixed	Full	Discrete	2D
Hanabi	cooperative	Partial	Discrete	1D
MATE	cooperative + mixed	Partial	Both	1D
GoBigger	cooperative + mixed	Both	Continuous	1D

Each environment has a readme file, standing as the instruction for this task, including env settings, installation, and important notes.

Initialize the algorithm

running target	api example
train & finetune	marl.algos.mappo(hyperparam_source=$ENV)
develop & debug	marl.algos.mappo(hyperparam_source="test")
3rd party env	marl.algos.mappo(hyperparam_source="common")

Here is a chart describing the characteristics of each algorithm:

algorithm	support task mode	discrete action	continuous action	policy type
IQL*