See parent class.

Perform a gradient update step. **Note**; The target update soft updates occur at the same frequency as the actor update frequencies.

Return the trainable variables. Parameters ---------- name : str the scope Returns ------- list of tf.Variable

Test the shape and bounds of an action or observation space. Parameters ---------- gym_space : gym.spaces.Box gym space object to

Compute the action space for the higher level policies. Parameters ---------- ob_space : gym.spaces.* the observation space of th

Test that an action or observation space is the correct size and bounds. Parameters ---------- gym_space : gym.spaces.Box gym spa

Reset the environment.

See parent class. This method initializes the target parameters to match the model parameters.

Add a new transition to the buffer. Parameters ---------- obs_t : Any the last observation action : array

Return the state indices for the intrinsic rewards. This assigns the indices of the state that are assigned goals, and subsequently rewarded

Return the observation of the agent. See States in the description of the environment for more.

Sample a batch of experiences. Returns ------- array_like batch of observations array_like ba

Get the standard deviation of a Tensor. Parameters ---------- tensor : tf.Tensor or tf.Variable the input tensor axis : int o

Determine whether a meta-policy should update its action. This is done by checking the length of the observation lists that are passe

Set the random seed for testing purposes.

Return, and potentially create, the environment. Parameters ---------- env : str or gym.Env the environment, or the name of a reg

Return the hyperparameters of a training algorithm from the parser.

Parse training options user can specify in command line. Parameters ---------- description : str the description of the script us

Perform a gradient update step. Parameters ---------- update_actor : bool specifies whether to update the actor p

Create a convolutional network. Parameters ---------- obs : tf.Variable the input to the model image_height : int the

Create a fully-connected neural network model. Parameters ---------- obs : tf.Variable the input to the model layers : list o

Return the negative euclidean distance between next_states and goals. Parameters ---------- states : array_like A (num_state_dims

See parent class.

Initialize the Ant Maze environment. Parameters ---------- use_contexts : bool, optional specifies whether to add

Return the processed observation dimension. If the context space is not None, it is included in the computation of this term.

Check whether the current level should be training. When using `pretrain_worker` the lowest level policy is trained every step, and h

Sample indices from a list.

Create a fully-connected layer. Parameters ---------- val : tf.Variable the input to the layer num_outputs : int numb

Print parameter shapes and number of parameters. Parameters ---------- scope : str scope containing the parameters param_type

Return the IDs of the currently observed and controlled RL vehicles. This is static in closed networks and dynamic in open networks.

Sample a batch of experiences. Returns ------- array_like (batch_size, obs_dim) batch of observations arr

Sample indices from a list.

Check if n_samples samples can be sampled from the buffer. Returns ------- bool True if enough sample exist, Fals

Check if n_samples samples can be sampled from the buffer. Returns ------- bool True if enough sample exist, Fals

See parent class.

Check whether the replay buffer is full or not. Returns ------- bool True if it is full, False otherwise

Advance the simulation by one step. Parameters ---------- velocities : array_like the action by the agent, define

Create a convolutional layer. Parameters ---------- val : tf.Variable the input to the layer filters : int the number

Render the environment state.

Update the index of individual agents. This auxiliary method helps supports assigning env_num variables when both computing actions a

Ensure that the directory specified exists, and if not, create it.

Get the observation from a (potentially unprocessed) variable. We assume multi-agent MADDPG style policies return a dictionary observations,

Check whether a policy is a goal-conditioned policy.

Test that the environment raises an Error in any param is missing. Parameters ---------- env_class : flow.envs.Env the environmen

Compute the L2 regularization penalty. Parameters ---------- l2_penalty : float L2 regularization penalty

Choose an initial state for all vehicles in the network. Parameters ---------- length : float the length of the r

Update the length of the ring road. Parameters ---------- length : float or [float, float] the length of the ring

Squash the output of the Gaussian distribution. This method also accounts for that in the log probability. The squashed mean is also returned

Compute log likelihood of a gaussian. Here we assume this is a Diagonal Gaussian. Parameters ---------- input_ : tf.Variable

Return the x,y position of the agent.

Perform the complete training operation. Parameters ---------- log_dir : str the directory where the training and

See parent class. We update the ring length to match a new value within a given range.

Create the soft and initial target updates. The initial model parameters are assumed to be stored under the scope name "model", while the tar

Compute the current step headway for all vehicles.

Perform the evaluation operation. This method runs the evaluation environment for a number of episodes and returns the cumulative rew

Return the processed observation. If the contextual term is not None, this will look as follows: -------

Add a new transition to the buffer. Parameters ---------- obs_t : array_like the last observation of an individua

Compute the environment reward. This is defined by the child classes.

Check whether a policy is for designed to support PPO.

Check whether a policy is for designed to support TRPO.

Create a critic tensor. Parameters ---------- obs : tf.compat.v1.placeholder the input observation placeholder

Reset the stored sample data-points.

Run a single training procedure. Parameters ---------- env : str or gym.Env the training/testing environment policy : type [

Set the x,y position of the agent.

Return the current step observation.

Return the current step observation.

Return the processed observation. If the contextual term is not None, this will look as follows: -------

Return the environment parameters of the fast ring environment. Parameters ---------- scale : int the scale of the ring environme

Log evaluation statistics. Parameters ---------- file_path : str path to the evaluation csv file start_ti

See parent class. Define which vehicles are observed for visualization purposes.

Clear internal memory that is used by the replay buffer.

Compute the RL acceleration from the desired acceleration. We reduce the decelerations at smaller speeds to smoothen the effects. Parameters

Import relevant data from each logging file in the specified folders. Parameters ---------- folders : list of str the path t

Check whether a policy is for designed to support SAC.

Check whether a policy is for designed to support TD3.

Load parameters for the replay buffer.

Create the critic variables. Parameters ---------- obs : tf.compat.v1.placeholder the input observation placehold

Create a critic tensor. Parameters ---------- obs : tf.compat.v1.placeholder the input observation placeholder

Create the critic variables. Parameters ---------- obs : tf.compat.v1.placeholder the input observation placehold

Plot the mean/std of the different models/algorithms. Parameters ---------- mean : list of np.ndarray a list of mean returns

Update a nested dictionary recursively recursively.

Get the variance of a Tensor. Parameters ---------- tensor : tf.Tensor the input tensor axis : int or list of int the

Sample a batch of experiences. An example for how a sample is collected from the list of observations/ actions for a three-level hier

Perform a gradient update step. This is done both at every level of the hierarchy. The kwargs argument for this method contains two

Perform gradient update step given a batch of data. Parameters ---------- obs0 : array_like batch of observations

See parent class.

Perform the sample collection operation over multiple steps. This method calls collect_sample for a multiple steps, and attempts to r

Compute the reward over a specific list of vehicles. Parameters ---------- rl_actions : array_like the actions pe

Compute the failsafe maximum acceleration. Parameters ---------- veh_ids : array_like the IDs of vehicles whose f

Return the full state observation.

Return the Ant observations.

Get the observation of the humanoid.

Compute the negative log-probability of an input action (x).

Get the actions from a given observation. Parameters ---------- obs : array_like the observation context

Sample a starting position for the agent.

Create the actor and critic variables for an individual agent. Parameters ---------- obs_ph : tf.compat.v1.placeholder

Create the actor and critic variables for an individual agent. Parameters ---------- obs_ph : tf.compat.v1.placeholder