Return my rank for the model parallel group.

Log a group of timers.

Split a tensor along its last dimension. Arguments: tensor: input tensor. num_partitions: number of partitions to split the tensor

If you're using outer_tokenizer, call `get_tokenizer(args, outer_tokenizer)` before `training_main`.

Create a linear module.

Create a linear module.

Create a linear module.

Create a linear module.

Create a linear module.

Create a linear module.

r""" Prepare the attention mask for the attention computation. Args: attention_mask (`torch.Tensor`): The

Drop any empty strings from the list of compile and link flags

Drop any empty strings from the list of compile and link flags

Drop any empty strings from the list of compile and link flags

Drop any empty strings from the list of compile and link flags

Drop any empty strings from the list of compile and link flags

r"""Fills the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distributi

Create a 1D, 2D, or 3D convolution module.

Create a 1D, 2D, or 3D convolution module.

Create a 1D, 2D, or 3D convolution module.

Create a 1D, 2D, or 3D convolution module.

Create a 1D, 2D, or 3D convolution module.

Create a 1D, 2D, or 3D convolution module.

get command token corresponding to `name`

Initialize model data parallel groups. Arguments: model_parallel_size: number of GPUs used to parallelize model. Let's say we h

r""" Normalize the encoder hidden states. Requires `self.norm_cross` to be specified when constructing the `Attention` class.

Split a tensor along its last dimension. Arguments: tensor: input tensor. num_partitions: number of partitions to split the tensor

Split a tensor along its last dimension. Arguments: tensor: input tensor. num_partitions: number of partitions to split the tensor

Sample a dataloader to sample from based on mixing probabilities. If one of the dataloaders is exhausted, we continue sampling from the other

Instantiate a PreTrainedBertModel from a pre-trained model file. Download and cache the pre-trained model file if needed.

Split a tensor along its last dimension. Arguments: tensor: input tensor. num_partitions: number of partitions to split the tensor

This method forwards all its arguments to LlamaTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring o

Get the model parallel group the caller rank belongs to.

Get the model parallel group the caller rank belongs to.

Get the model parallel group the caller rank belongs to.

Get the model parallel group the caller rank belongs to.

Get the model parallel group the caller rank belongs to.

Make a standard normalization layer. :param channels: number of input channels. :return: an nn.Module for normalization.

Save a model checkpoint.

Init method based on N(0, sigma).

Initialize model data parallel groups. Arguments: model_parallel_size: number of GPUs used to parallelize model. Let's say we h