Loss function. This function computes the loss (negative log-likelihood). Parameters ---------- log_probas (`torch.FloatTensor` of shape `(batch_size, sequence_length, vocabulary_size)`) A tensor containing the log-probabilities of the next token for
(self, log_probas, targets, mask)
| 367 | return outputs |
| 368 | |
| 369 | def loss(self, log_probas, targets, mask): |
| 370 | """Loss function. |
| 371 | |
| 372 | This function computes the loss (negative log-likelihood). |
| 373 | |
| 374 | Parameters |
| 375 | ---------- |
| 376 | log_probas (`torch.FloatTensor` of shape `(batch_size, sequence_length, vocabulary_size)`) |
| 377 | A tensor containing the log-probabilities of the next token for |
| 378 | all positions in each sequence of the batch. |
| 379 | |
| 380 | targets (`torch.LongTensor` of shape `(batch_size, sequence_length)`) |
| 381 | A tensor containing the target next tokens for all positions in |
| 382 | each sequence of the batch. |
| 383 | |
| 384 | mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`) |
| 385 | A tensor containing values in {0, 1} only, where the value is 0 |
| 386 | for positions corresponding to padding in the sequence, and 1 |
| 387 | otherwise. |
| 388 | |
| 389 | Returns |
| 390 | ------- |
| 391 | loss (`torch.FloatTensor` scalar) |
| 392 | The scalar loss, corresponding to the (mean) negative log-likelihood. |
| 393 | """ |
| 394 | |
| 395 | # ========================== |
| 396 | loss = nn.NLLLoss(reduction='none')(log_probas.view(-1, log_probas.size(-1)), |
| 397 | targets.view(-1)) |
| 398 | |
| 399 | masked_loss = loss * mask.view(-1) |
| 400 | mean_loss = masked_loss.sum() / mask.sum() |
| 401 | # ========================== |
| 402 | return mean_loss |
| 403 | |
| 404 | @classmethod |
| 405 | def load_embeddings_from( |