Converts an id to a token, special tokens included

Converts an index (integer) in a token (str) using the vocab.

Converts an index (integer) in a token (str) using the vocab.

Converts a token to an id using the vocab, special tokens included

Converts a token (str) in an id using the vocab.

Converts a token (str) in an id using the vocab.

Simple forward step with cross-entropy loss.

Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based vocabulary or sub-words for su

Returns a tokenized string.

Add token-type embedding. This function is provided so we can add token-type embeddings in case the pretrained model does not have it.

Arguments: x: (batch_size, seqlen, nheads, headdim) if cu_seqlens is None else (total_seqlen, nheads, headdim) cos, s

x: (batch_size, seqlen, nheads, headdim) cos, sin: (seqlen, rotary_dim / 2) or (batch_size, seqlen, rotary_dim / 2)

Compute and scale the gradient for auxiliary loss.. Args: grad_output (torch.Tensor): The gradient of the output. Return

Run beam search and post-process outputs, i.e., detokenize, move to cpu and convert to list. Args: model (torch.nn.Module): The m

Build train, valid, and test datasets for pretraining a LLAMA model on mmap format data. Args: data_prefix (str): common prefix added