Method __init__

numpy_ml/rl_models/agents.py:794–899

r""" A temporal difference learning agent with expected SARSA (on-policy) [3]_ or TD(0) `Q`-learning (off-policy) [4]_ updates. Notes ----- The expected SARSA on-policy TD(0) update is: .. math:: Q(s, a) \leftarrow Q(s, a) + \eta \left(

(
        self,
        env,
        lr=0.4,
        epsilon=0.1,
        n_tilings=8,
        obs_max=None,
        obs_min=None,
        grid_dims=[8, 8],
        off_policy=False,
        temporal_discount=0.99,
    )

class TemporalDifferenceAgent(AgentBase):
    def __init__(
        self,
        env,
        lr=0.4,
        epsilon=0.1,
        n_tilings=8,
        obs_max=None,
        obs_min=None,
        grid_dims=[8, 8],
        off_policy=False,
        temporal_discount=0.99,
    ):
        r"""
        A temporal difference learning agent with expected SARSA (on-policy) [3]_ or
        TD(0) `Q`-learning (off-policy) [4]_ updates.

        Notes
        -----
        The expected SARSA on-policy TD(0) update is:

        .. math::

            Q(s, a) \leftarrow Q(s, a) + \eta \left(
                r + \gamma \mathbb{E}_\pi[Q(s', a') \mid s'] - Q(s, a)
            \right)

        and the TD(0) off-policy Q-learning upate is:

        .. math::

            Q(s, a) \leftarrow Q(s, a) + \eta (
                r + \gamma \max_a \left\{ Q(s', a) \right\} - Q(s, a)
            )

        where in each case we have taken action `a` in state `s`, received
        reward `r`, and transitioned into state :math:`s'`. In the above
        equations, :math:`\eta` is a learning rate parameter, :math:`\gamma` is
        a temporal discount factor, and :math:`\mathbb{E}_\pi[ Q[s', a'] \mid
        s']` is the expected value under the current policy :math:`\pi` of the
        Q function conditioned that we are in state :math:`s'`.

        Observe that the expected SARSA update can be used for both on- and
        off-policy methods. In an off-policy context, if the target policy is
        greedy and the expectation is taken wrt. the target policy then the
        expected SARSA update is exactly Q-learning.

        NB. For this implementation the agent requires a discrete action
        space, but will try to discretize the observation space via tiling if
        it is continuous.

        References
        ----------
        .. [3] Rummery, G. & Niranjan, M. (1994). *On-Line Q-learning Using
           Connectionist Systems*. Tech Report 166. Cambridge University
           Department of Engineering.
        .. [4] Watkins, C. (1989). Learning from delayed rewards. *PhD thesis,
           Cambridge University*.