Check if a model name exists

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Unfreeze parameters of the specified modules and those of all their hierarchical descendants. This is done in place. Args: root_modul

Feature map up-sampling.

Args: cfg (dict): Model architecture configuration in_chans (int): Number of input channels (default: 3)

Args: in_chans: Number of input image channels. num_classes: Number of classes for classification head. g

Args: img_size: patch_size: in_chans: num_classes: global_pool: embe

Args: cfg (RegNetCfg): Model architecture configuration in_chans (int): Number of input channels (default: 3)

Method to fuse batchnorm layer with preceeding conv layer. Reference: https://github.com/DingXiaoH/RepVGG/blob/main/repvgg.py#L95 Arg

Method to fuse batchnorm layer with preceeding conv layer. Reference: https://github.com/DingXiaoH/RepVGG/blob/main/repvgg.py#L95

Method to fuse batchnorm layer with preceeding conv layer. Reference: https://github.com/DingXiaoH/RepVGG/blob/main/repvgg.py#L95

FBNetV3 Paper: `FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining` - https://arxiv.org/abs/2006.02049 FIXME untes

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): Th

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): Th

Find the latest checkpoint in the save path.

A func to sync embedding input with alltoall in sequence parallel gather sequence dimension and scatter head dim: e.g. seq_dim: 1, head_d

Gets all parameters in all param_groups with gradients

Get a specific model default_cfg value by key. None if key doesn't exist.

Create a huggingface pretrained processor.

Get the resource pool of the worker.

Create a huggingface pretrained tokenizer.

Get ulysses sequence parallel world size.

Select a subset of the DataProto via index. Args: index (list, ndarray, torch.Tensor): a list of indices to select. Retu