Stacks n steps into single numpy arrays Args: experience: (type: dictionary) subtrajectory information

Perform update on target model networks. This update is either 'soft', meaning the target model drifts towards the current model at

Computes the loss with current data. also returns a dictionary of statistics which includes value loss and explained variance

Method that makes baseline function updates. calls _value_loss method Note: self.clip_actor_gradient determines whether gradient is

Abstract method for taking actions. You should check `self.agent_mode` in the function and change act() logic with respect to

Spin up a multi-node distributed Surreal experiment. Put any command line args that pass to the config script after "--"

qu = _ScoreQueue(3) for i in [1,3,2,4,5,7,3, 8, 2, 2, 1]: print(qu.add(i*10, 'path'+str(i)), qu._queue)

Args: hash_dict: {obs_hash: [ .. can be nested .. ]} nonhash_dict: {reward: -1.2, done: True, ...}

Capitalize an Atari game name properly

This function defines what attributes should be serialized when we are saving a checkpoint. See implementations

Method that clears all gradients from all the parameters from critic, cnn_stem (optional), and rnn_stem (optional)

Dump content into binary Returns: bytes

Actively evict old experiences.

join file paths and expand special symbols like `~` for home dir

Fetch the metadata of parameters on parameter server Returns: dictionary of metadata

Called by agent to retrieve parameters By default, pulls from PS ONLY WHEN the parameter hash changes to prev

Find the highest index `i` in the array such that sum(arr[0] + arr[1] + ... + arr[i - i]) <= prefixsum if array values are probabi

Serialized all currenct content of the buffer into binary Returns: binary data of (exp_list, ob_storage)

forward pass critic to generate policy with option to use z-filter also returns an updated LSTM hidden/cell state when necess

Returns the agent_ids corresponding to batch_id Args: batch_id: index of batch Returns: agent_i

Args: enum_class: option: if the value doesn't belong to Enum, throw error. Can be either the str name or the actual

Returns a subclass of EnvBase, created from self.env_config

Returns the eval_ids corresponding to batch_id Args: batch_id: index of batch Returns: eval_ids

Actual experiment folder will be <mount_path>/<root_subfolder>/<experiment_name>/

returns score_list, filepath_list

Get host name, alias with ``socket.gethostname()``

Args: - prelinear_size: size of feature vector before the linear layers, like flattened conv or LSTM features

Runs load balancing proxy thread and self.shards ParameterServer processes Returns after all threads and proc

Abstract method runs one step of learning Args: batch_exp: batched experience, format is a list of whatever

Method computes loglikelihood of action (a) given probability (prob)

The main function to be called ``` if __name__ == '__main__': launcher = Launcher() launcher.main()

Main loop ran by the agent script Override if you want to customize agent behavior completely

Main loop ran by the eval script Override if you want to customize eval behavior completely

One loop of learner, runs one learn operation of learner

Setup before constant looping

Method deterministically sample actions of maximum likelihood

Dict of modules to be broadcasted to the parameter server. MUST be consistent with the agent's `module_dict()`

Returns: a dict of name -> surreal.utils.pytorch.Module

Called when a new parameter is fetched.

Called after act is called by agent main script

Called by agent process. Can beused to reset internal states after an episode ends I.e. after the post_action whe

Called before act is called by agent main script

Called by agent process. Can beused to reset internal states before an episode starts

Applies custom wrapper to the environment as necessary Args: @env: subclass of EnvBse Returns: @

Applies custom wrapper to the environment as necessary Only changes agent behavior

Applies custom wrapper to the environment as necessary Only changes eval behavior

Restores an agent from a model.