Method _simulate_behavior

numpy_ml/rl_models/agents.py:1678–1715

Perform random-sample one-step tabular Q-planning with prioritized sweeping. Notes ----- This approach uses a priority queue to retrieve the state-action pairs from the agent's history with largest change to their Q-values if backed up. When

(self)

        return priority

    def _simulate_behavior(self):
        """
        Perform random-sample one-step tabular Q-planning with prioritized
        sweeping.

        Notes
        -----
        This approach uses a priority queue to retrieve the state-action pairs
        from the agent's history with largest change to their Q-values if
        backed up. When the first pair in the queue is backed up, the effect on
        each of its predecessor pairs is computed. If the predecessor's
        priority is greater than a small threshold the pair is added to the
        queue and the process is repeated until either the queue is empty or we
        have exceeded a `n_simulated_actions` updates.
        """
        env_model = self.parameters["model"]
        sweep_queue = self.derived_variables["sweep_queue"]
        for _ in range(self.n_simulated_actions):
            if len(sweep_queue) == 0:
                break

            # select (s, a) pair with the largest update (priority)
            sq_items = list(sweep_queue.items())
            (s_sim, a_sim), _ = sorted(sq_items, key=lambda x: x[1], reverse=True)[0]

            # remove entry from queue
            del sweep_queue[(s_sim, a_sim)]

            # update Q function for (s_sim, a_sim) using the full-backup
            # version of the TD(0) Q-learning update
            self._update(s_sim, a_sim)

            # get all (_s, _a) pairs that lead to s_sim (ie., s_sim's predecessors)
            pairs = env_model.state_action_pairs_leading_to_outcome(s_sim)

            # add predecessors to queue if their priority exceeds thresh
            for (_s, _a) in pairs:
                self._update_queue(_s, _a)

    def _update(self, s, a):
        """