Method __init__

numpy_ml/rl_models/agents.py:1305–1426

r""" A Dyna-`Q` / Dyna-`Q+` agent [5]_ with full TD(0) `Q`-learning updates via prioritized-sweeping [6]_ . Notes ----- This approach consists of three components: a planning method involving simulated actions, a direct RL method where the agent direc

(
        self,
        env,
        lr=0.4,
        epsilon=0.1,
        n_tilings=8,
        obs_max=None,
        obs_min=None,
        q_plus=False,
        grid_dims=[8, 8],
        explore_weight=0.05,
        temporal_discount=0.9,
        n_simulated_actions=50,
    )

class DynaAgent(AgentBase):
    def __init__(
        self,
        env,
        lr=0.4,
        epsilon=0.1,
        n_tilings=8,
        obs_max=None,
        obs_min=None,
        q_plus=False,
        grid_dims=[8, 8],
        explore_weight=0.05,
        temporal_discount=0.9,
        n_simulated_actions=50,
    ):
        r"""
        A Dyna-`Q` / Dyna-`Q+` agent [5]_ with full TD(0) `Q`-learning updates via
        prioritized-sweeping [6]_ .

        Notes
        -----
        This approach consists of three components: a planning method involving
        simulated actions, a direct RL method where the agent directly interacts
        with the environment, and a model-learning method where the agent
        learns to better represent the environment during planning.

        During planning, the agent performs random-sample one-step tabular
        Q-planning with prioritized sweeping. This entails using a priority
        queue to retrieve the state-action pairs from the agent's history which
        would stand to have the largest change to their Q-values if backed up.
        Specifically, for state action pair `(s, a)` the priority value is:

        .. math::

            P = \sum_{s'} p(s') | r + \gamma \max_a \{Q(s', a) \} - Q(s, a) |

        which corresponds to the absolute magnitude of the TD(0) Q-learning
        backup for the pair.

        When the first pair in the queue is backed up, the effect on each of
        its predecessor pairs is computed. If the predecessor's priority is
        greater than a small threshold the pair is added to the queue and the
        process is repeated until either the queue is empty or we have exceeded
        `n_simulated_actions` updates. These backups occur without the agent
        taking any action in the environment and thus constitute simulations
        based on the agent's current model of the environment (i.e., its
        tabular state-action history).

        During the direct RL phase, the agent takes an action based on its
        current behavior policy and Q function and receives a reward from the
        environment. The agent logs this state-action-reward-new state tuple in
        its interaction table (i.e., environment model) and updates its Q
        function using a full-backup version of the Q-learning update:

        .. math::

            Q(s, a) \leftarrow Q(s, a) + \eta \sum_{r, s'} p(r, s' \mid s, a)
                \left(r + \gamma \max_a \{ Q(s', a) \} - Q(s, a) \right)