Source code modified from EasyDict to support extra methods. Does not support subclass attribute any more. https://github.com/makinacorpu

Two-step serialization. 1. Each element's state_dict() is called 2. The overall dict is then pickled.

PPO Model class that wraps aroud the actor and critic networks Attributes: actor: Actor network, see surreal.model.model_

Standalone script for PS node that runs in an infinite loop. The ParameterServer subscribes to learner to get the latest

Class that encapsulates Diagonal Gaussian Probability distribution Attributes: d: action dimension Member Functio

Given the inputs allowed in env_config.observation, reject any inputs not specified in the config.

Simple wrapper to limit maximum steps of an environment

Keeps track of average of things Always caches the latest value so no division by 0

Dummy wrapper for deepmind control environment using pixels. The output of observation_spec and action_spec will match the output for a dm_c

Temporarily holds and deduplicates experience

`send()` logic can be overwritten to support more complicated agent experiences, such as multiagent, self-play, etc.

Base class for all classes that send experience in format { 'obs': [state_1, ..., state_n] 'obs_next': [st

Sends observations in format { 'obs': [state, next_state] 'action': action, 'reward': reward,

Accepts experience from agents, deduplicates experience whenever possible

Accepts experience sent by SSAR experience senders

Convenience class that initializes everything from session config + batch_size Fetches data from replay in multiple processe

Accepts input by ExpSenderWrapperMultiStepMovingWindowWithInfo aggregate() returns float Tensors { obs = batch_si

PPO custom actor network structure

PPO custom critic network structure

On agent side, sends requests to parameter servers to fetch the latest parameters.

Publishes parameters from the learner side Using ZmqPub socket

Keeps historical average of rewards Attributes: eps: tolerance value for computing reward filter (whitening)

Accepts experience sent by SSAR experience senders aggregate() returns float arrays: { obs = batch_size * observa

Runs multiple parameter servers in parallel

https://docs.python.org/3.4/library/enum.html#duplicatefreeenum The created options will automatically have the same string value as name.

Manages the following: 1) Figure out what docker images need to be built 2) Build (and push) them

A tensorplex client that aggregates output from multiple threads Thread safe Update criterion: 1) A global step is p

Environment Wrappers for automatically rendering and saving test runs. Attributes: public attributes env (Env): environme

Keeps historical average and std of inputs Whitens data and clamps to +/- 5 std Attributes: in_size: state dimens

Reverse score sorting

Important: When extending this class, make sure to follow the init method signature so that orchestrating functions can properly ini

Call the special method ._initialize() after __init__. Useful if some logic must be run after the object is constructed. For example, the

Periodically saves attribute of a learner or agent class. For PyTorch modules, must extend surreal.utils.pytorch.Module instead of the bu

DDPGAgent: subclass of Agent that contains DDPG algorithm logic Attributes: model: A DDPG neural network model to generate actions fr