Assumes self._history contains the last n frames from the environment Concatenates the frames together along the depth axis

Args: tag_value_dict: global_step: None to use the internal counter Returns: - None if period i

Args: exp_list: Returns: aggregated experience

Low collision hash of any binary string For designating the 16-char object key in Redis. Runs at 200 mu-second per hash on Macbook pro.

Method that clears all gradients from all the parameters from actor, cnn_stem (optional), and rnn_stem (optional)

Method computes entropy of a given probability (prob)

Method that returns generator that contains all the parameters from critic, cnn_stem (optional), and rnn_stem (optional)

Returns: True if the sequence is a collections.Sequence and not a string.

Called when connection with parameter server fails to be established

Called when connection with parameter server is succesfully established

Will only save at the end of each period Args: global_steps: the number of iterations score: the evaluation

Called by learner. Publishes model parameters with additional info Args: iteration: current learning iteration

Learner publishes latest parameters to the parameter server. Args: iteration: the current number of learning iterations

Returns result of applying `self.operation` to a contiguous subsequence of the array. self.operation(arr[start], operation(arr[sta

reset of LSTM hidden and cell states

Args: target: can be one of the following semantics - int: 0 for the last (or best), 1 for the second last (or best

This function is called in _sample_handler for learner side Zmq request Args: batch_size: Returns:

Saves checkpoint to disk Args: file_path: locatioin to save

Same as launch_agent, but instead of running agent.main() infinite loop, returns the agent instance Args: agent

Same as run_learner, but returns the Learner instance instead of calling learner.main_loop()

Returns: True if we enter the next period

updates kept parameters to that of another model Args: net: another PPO_Model instance

Args: folder: checkpoint folder path name: prefix of the checkpoint tracked_obj: arbitrary object whose c

Computes the loss with current data. also returns a dictionary of statistics which includes surrogate loss, clipped surrogate los, po

Method that makes policy updates. calls _adapt_loss method Note: self.clip_actor_gradient determines whether gradient is clipped

Computes the loss with current data. also returns a dictionary of statistics which includes surrogate loss, clipped surrogate los, po

Method that makes policy updates. calls _clip_loss method Note: self.clip_actor_gradient determines whether gradient is clipped

Finds the .py file corresponding to the algorithm specified Args: name: ddpg / ppo / <path in container to compatibl

Finds the .py file corresponding to the algorithm specified Args: name: ddpg / ppo / <path in container to compatibl

Finds the .py file corresponding to the algorithm specified Args: name: ddpg / ppo / <path in container to compatibl

computes generalized advantage estimate and corresponding N-step return. Details of algorithm can be found here: https://arxiv.org/p

Gathers corresponding action informations from partial trajectories Args: experience: (type: dictionary) subt

https://docs.python.org/3/library/inspect.html#inspect.BoundArguments def f(a, b, c=5, d=6): pass get_bound_args(f, 3, 6, d=100) -> {'a':

Get the periodic tensorplex object Args: @name: The name of the collection of metrics

initializes exploration noise and populates self.noise, a callable that returns noise of dimension same as action

implements AutoInitializeMeta meta class. self.module_dict can only happen after the module is constructed by subclasses.

Note that while the replay contains uint8, the aggregator returns float32 tensors Arguments: obs: an observation

main method for optimization that calls _adapt/clip_update and _value_update epoch_policy and epoch_baseline times respectiv

function that manages metrics and behavior after parameter release Actions include: adjusts adaptive thresho

Loading experiences from numpy to torch.FloatTensor type Args: batch: BeneDict of experiences containing fol

experiment_name will be used as DNS, so must not have underscore or dot

Args: exp: a nested dict or list Only dict keys that end with `_hash` will be retrieved. The proc

This is a helper function to sample expriences with probabilities proportional to their priorities. Returns a list of indice

Setup the necessary environment variables

Assigns GPU to agents and learners in an optimal way. No GPU, do nothing

Assigns GPU to agents and learners in an optimal way. No GPU, do nothing

Creates tensorplex logger and loggerplex logger Initializes bookkeeping values