Compute mean of tensor with a masked values.

See http://www.codinghorror.com/blog/archives/001018.html

Override model level attn/self-attn/block kwargs w/ block level NOTE: kwargs are NOT merged across levels, block_kwargs will fully replace model

Pad a nD tensors in the last dim to max_seq_len.

Note that this function only rename the key in the batch

Resize relative position bias table using more advanced interpolation. Modified from code in Microsoft Unilm (https://github.com/microsoft/unilm

Restore a batch from dynamic batching. Args: data (torch.Tensor): The input data. batch_idx_list (List[List[int]]): The list

Pad and slice input_ids to be divisible by sp_size Pad position_ids to be divisible by sp_size. Note both input_ids_rmpad and position_i

Args: block (nn.Module): class for the residual block. Options are BasicBlock, Bottleneck. layers (List[int]) : numbe

Do not add eos again if user already added it.

Crops to center of image with padding then scales image_size.

expand a 'base_filter' to 'base_filter.*' if no tag portion

Returns: log_probs: # (bs, response_len)

Freeze or unfreeze parameters of the specified modules and those of all their hierarchical descendants. This is done in place. Args:

Method to fuse batchnorm layer with conv layer. Args: conv: Convolutional kernel weights. bn: Batchnorm 2d layer.

Creates an EfficientNet-V2 Large model Ref impl: https://github.com/google/automl/tree/master/efficientnetv2 Paper: `EfficientNetV2: Smaller

Creates an EfficientNet-V2 Medium model Ref impl: https://github.com/google/automl/tree/master/efficientnetv2 Paper: `EfficientNetV2: Smalle

Creates an EfficientNet-V2 Xtra-Large model Ref impl: https://github.com/google/automl/tree/master/efficientnetv2 Paper: `EfficientNetV2: Sm

Creates a MixNet Small model. Ref impl: https://github.com/tensorflow/tpu/tree/master/models/official/mnasnet/mixnet Paper: https://arxiv.org

Return True if scheduler in noise range.

Lazily initialize worker (in worker processes)

Lazily initialize worker (in worker processes)

Lazily initialize the dataset. This is necessary to init the Tensorflow dataset pipeline in the (dataloader) process that will be us

Method generates the attention mask used in shift case.

Method initializes the pair-wise relative positions to compute the positional biases.

Per-stage depth scaling Scales the block repeats in each stage. This depth scaling impl maintains compatibility with the EfficientNet scaling