Assume that arguments are contiguous and aligned to 16 bytes dx == None means that it was a post-norm architecture (x = drop(x0) + residual wa

Assume that arguments are contiguous and aligned to 16 bytes

Assume that arguments are contiguous and aligned to 16 bytes dx == None means that it was a post-norm architecture (x = drop(x0) + residual wa

Assume that arguments are contiguous and aligned to 16 bytes

Assume that arguments are contiguous and aligned to 16 bytes dx == None means that it was a post-norm architecture (x = drop(x0) + residual wa

Assume that arguments are contiguous and aligned to 16 bytes

Returns the kwargs needed to apply `accelerate` in `AutoModel.from_pretrained`.

Helper method during initialization. Determines the backend ("causal" (decoder-only) or "seq2seq" (encoder-decoder)) model ty

Creates an dictionary of tasks index with the following metadata, - `type`, that can be either `task`, `python_task`, or `group`.

Create the state if it doesn't exist, zero it out otherwise. Do this recursively if the layer_idx is a tuple.

Instantiate a single decoder

get logits for the entire sequence :param input_ids: torch.Tensor A torch tensor of shape [batch, sequence_cont]

Lower text and remove punctuation, articles and extra whitespace.

Based on number of samples and sequence length, calculate how many epochs will be needed.

Total number of tokens in the dataset.

Process logprobs and tokens. :param tokens: list Input tokens (potentially left-truncated) :param outputs: RequestOutput

custom init strategy

Modify the return value of the collate_fn. Assign a name to each element of the returned tuple beyond the (x, y) pairs See In

kv: (batch_size, seqlen, 2, nheads, head_dim) or (batch_size, 1, 2, nheads, head_dim)

kv: (batch_size, seqlen, 2, nheads, head_dim) or (batch_size, 1, 2, nheads, head_dim)

kv: (batch_size, seqlen, 2, nheads, head_dim) or (batch_size, 1, 2, nheads, head_dim)

kv: (batch_size, seqlen, 2, nheads, head_dim) or (batch_size, 1, 2, nheads, head_dim)

:returns: {str: [metric_score] -> float} A dictionary where keys are the names of submetrics and values are functions

Creates a state tensor of shape (b, d, k)

Wrapper function around the Anthropic completion API client with exponential back-off in case of RateLimitError. params: client: anth

Iterates over FilterEnsembles and applies them to instances

Call torch.distributed.barrier() if distritubed is in use

Call torch.distributed.barrier() if distritubed is in use

Assume that y has shape (B, D) and bias has shape (D)

Build a set of Instances for a task, and store them in task.instances

Build train/valid/test datasets.

Copy current averaged parameters into given collection of parameters. Args: parameters: Iterable of `torch.nn.Parameter`;

Used for byte-level perplexity metrics in rolling loglikelihood

Downstream loglikelihood_rolling perplexity tasks with custom word boundaries should override this!

Downstream tasks with custom word boundaries should override this!