LlamaRMSNorm is equivalent to T5LayerNorm

LlamaRMSNorm is equivalent to T5LayerNorm

Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.

Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.

Make causal mask used for bi-directional self-attention.

Make causal mask used for bi-directional self-attention.

Tokenize a list of strings.

Resize tokenizer and embedding. Note: This is the unoptimized version that may make your embedding size not be divisible by 64.

calculate chunk wise PPL, and average across chunked examples

calculate LLM accuracy and PPL. GPU always OOM as logits is large

r""" Args: labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing

r""" labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression

r""" Args: labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing

r""" labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression

Preprocess the data by tokenizing.