given split proportions checks if should split Examples: >>> should_split([10,0,0]) False >>> should_split([1,.1,.2]) True

Save the current parameters for restoring later. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be

Runs basic whitespace cleaning and splitting on a piece of text.

RMSNorm is equivalent to T5LayerNorm

extracts samples only pertaining to this worker's batch

Gather tensors and concatinate along the last dimension.

All-reduce the the input tensor across model parallel group.

Split the tensor along its last dimension and keep the corresponding slice.

Train/valid/test data arguments.

Model arguments

Training arguments.

Appends dimensions to the end of a tensor until it has target_dims dimensions.

Arguments: x: (batch, nheads, seqlen, headdim) if cu_seqlens is None else (total_seqlen, nheads, headdim). cos: (seql

Returns nvcc compute capability compile flags. 1. `TORCH_CUDA_ARCH_LIST` takes priority over `cross_compile_archs`. 2. If ne

Convert Bytes Converts bytes to scaled format (e.g KB, MB, etc.)

Converts a sequence of tokens into ids using the vocab.

A counter for the `thop` package to count the operations in an attention operation. Meant to be used like: macs, params = thop.pr

Calculate the elapsed time.

@param text: Text to encode. @param linebreak: Whether to encode newline (\n) in text. @param whitespaces: Whether to encode

Ensure that numerator is divisible by the denominator.

Helper function to evaluate and dump results on screen.