Class RNN

imperative/python/megengine/module/rnn.py:308–428

r"""Applies a multi-layer Elman RNN with :math:`\tanh` or :math:`\text{ReLU}` non-linearity to an input sequence. For each element in the input sequence, each layer computes the following function: .. math:: h_t = \tanh(W_{ih} x_t + b_{ih} + W_{hh} h_{(t-1)} + b_{hh}) whe

class RNN(RNNBase):

    r"""Applies a multi-layer Elman RNN with :math:`\tanh` or :math:`\text{ReLU}` non-linearity to an
    input sequence.


    For each element in the input sequence, each layer computes the following function:

    .. math::
        h_t = \tanh(W_{ih} x_t + b_{ih} + W_{hh} h_{(t-1)} + b_{hh})

    where :math:`h_t` is the hidden state at time `t`, :math:`x_t` is
    the input at time `t`, and :math:`h_{(t-1)}` is the hidden state of the
    previous layer at time `t-1` or the initial hidden state at time `0`.
    If :attr:`nonlinearity` is ``'relu'``, then :math:`\text{ReLU}` is used instead of :math:`\tanh`.

    Args:
        input_size(:class:`int`): The number of expected features in the input `x`.
        hidden_size(:class:`int`): The number of features in the hidden state `h`.
        num_layers(:class:`int`): Number of recurrent layers. E.g., setting ``num_layers=2``
            would mean stacking two RNNs together to form a `stacked RNN`,
            with the second RNN taking in outputs of the first RNN and
            computing the final results. Default: 1.
        nonlinearity(:class:`str`): The non-linearity to use. Can be either ``'tanh'`` or ``'relu'``. Default: ``'tanh'``.
        bias(:class:`bool`): If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`.
            Default: ``True``.
        batch_first(:class:`bool`): If ``True``, then the input and output tensors are provided
            as `(batch, seq, feature)` instead of `(seq, batch, feature)`.
            Note that this does not apply to hidden or cell states. See the
            Inputs/Outputs sections below for details.  Default: ``False``.
        dropout(:class:`float`): If non-zero, introduces a `Dropout` layer on the outputs of each
            RNN layer except the last layer, with dropout probability equal to
            :attr:`dropout`. Default: 0.
        bidirectional(:class:`bool`): If ``True``, becomes a bidirectional RNN. Default: ``False``.

    Shape:
        - Inputs: input, h_0
            input: :math:`(L, N, H_{in})` when ``batch_first=False`` or :math:`(N, L, H_{in})` 
                when ``batch_first=True``. Containing the features of the input sequence.
            h_0: :math:`(D * \text{num\_layers}, N, H_{out})`. Containing the initial hidden
                state for each element in the batch. Defaults to zeros if not provided.

            where:

            .. math::
                \begin{aligned}
                    N ={} & \text{batch size} \\
                    L ={} & \text{sequence length} \\
                    D ={} & 2 \text{ if bidirectional=True otherwise } 1 \\
                    H_{in} ={} & \text{input\_size} \\
                    H_{out} ={} & \text{hidden\_size}
                \end{aligned}

        - Outputs: output, h_n
            output: :math:`(L, N, D * H_{out})` when ``batch_first=False`` or :math:`(N, L, D * H_{out})` when ``batch_first=True``. 
                Containing the output features `(h_t)` from the last layer of the RNN, for each `t`.
            h_n: :math:`(D * \text{num\_layers}, N, H_{out})`. Containing the final hidden state for each element in the batch.