Method forward

Language_Model/GPT1.py:339–367

Mini GPT-1. This is a small version of OpenAI's GPT-1 transformer for (causal) language modeling. This module returns for each position in the sequence the log-probabilities of the next token. Parameters ---------- inputs (`torch.LongTensor` of shape

(self, inputs)

        return embeddings

    def forward(self, inputs):
        """Mini GPT-1.

        This is a small version of OpenAI's GPT-1 transformer for (causal)
        language modeling. This module returns for each position in the
        sequence the log-probabilities of the next token.

        Parameters
        ----------
        inputs (`torch.LongTensor` of shape `(batch_size, sequence_length)`)
            The input tensor containing the token sequences.

        Returns
        -------
        log_probas (`torch.FloatTensor` of shape `(batch_size, sequence_length, vocabulary_size)`)
            A tensor containing the log-probabilities of the next token for
            all positions in each sequence of the batch. For example, `log_probas[0, 3, 6]`
            corresponds to log p(x_{5} = token_{7} | x_{0:4}) (x_{5} for the word
            after x_{4} at index 3, and token_{7} for index 6) for the 1st sequence
            of the batch (index 0).
        """

        # ==========================
        x = self.get_embeddings(inputs)
        for layer in self.layers:
            x = layer(x)
        outputs = nn.LogSoftmax(dim=-1)(self.classifier(x))
        # ==========================
        return outputs

    def loss(self, log_probas, targets, mask):
        """Loss function.