Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`] Args: config: LlamaConfig

A single transformer layer. Transformer layer takes input with size [s, b, h] and returns an output of the same size.

MLP. MLP will take the input with h hidden state, project it to 4*h hidden dimension, perform nonlinear transformation, and project the s

Parallel self-attention layer abstract class. Self-attention layer takes input with size [s, b, h] and returns output of the same size.

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

Language model embeddings.

Transformer class.

Multi-headed attention from 'Attention Is All You Need' paper

LlamaRotaryEmbedding extended with Dynamic NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla

Llama flash attention module. This module inherits from `LlamaAttention` as the weights of the module stays untouched. The only required chan

LlamaRotaryEmbedding extended with linear scaling. Credits to the Reddit user /u/kaiokendev

Llama attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from `LlamaAttention` as the weights of t