Initialize the weights.

Initialize the weights

Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel

Setup the optimizer. We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the

Forward pass through the OpenELMRMSNorm layer. Args: x (torch.Tensor): The input tensor. Returns: tor

The forward function of RoPE embeddings. Args: query: Query embeddings in the transformer model. The shape of query embed

Forward pass of multi-head self-attention. Args: hidden_states: Input tensor of the shape [batch size, sequence length, m

Forward function of FFN layer. Args: x: Input tensor of the shape [batch size, sequence length, model dimension]. Returns:

Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (

r""" encoder_hidden_states (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`): Seq

r""" encoder_hidden_states (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`): Seq

r""" labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Labels for computing the masked l

Args: x (torch.Tensor): image features shape (b, T, n1, D) latent (torch.Tensor): latent features

Calculate the shape of the image patch grid after the preprocessing for images of any resolution. Args: image_size (tuple): The size