This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

Rotates half the hidden dims of the input.

Converts an index (integer) in a token (str) using the vocab.

dynamic RoPE layers should recompute `inv_freq` in the following situations: 1 - growing beyond the cached sequence length (allow sca

Enhanced Transformer with Rotary Position Embedding. Derived from: https://github.com/labmlai/annotated_deep_learning_paper_implementations/b

Split a tensor along its last dimension. Arguments: tensor: input tensor. num_partitions: number of partitions to split the tenso

LlamaRMSNorm is equivalent to T5LayerNorm

Converts a token (str) in an id using the vocab.

Initialize the weights.

Pad encoded inputs (on left/right and up to predefined length or max length in the batch) Args: encoded_inputs:

This function is used to re-order the `past_key_values` cache if [`~PreTrainedModel.beam_search`] or [`~PreTrainedModel.beam_sample`]

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens

Converts a sequence of tokens in a single string.