Load a config from a wandb run ID. Parameters: run_id (str): A full wandb run id like "hazy-research/attention/159o6asi"

Load a model from a wandb run ID. Parameters: run_id (str): A full wandb run id like "hazy-research/attention/159o6asi"

Loads a dictionary of task objects from a list :param task_list: Union[str, list] = None Single string or list of string of task

One issue is that this can only be run with trainer.devices=1 right now. So restart with that after training completes. Add resume=Tr

Apply repetition penalty. See https://arxiv.org/abs/1909.05858 logits: (batch_size, vocab_size) prev_output_tokens: (batch_size, seq_len)

Generate orthogonal Gaussian random features

Override the default metrics used for evaluation with custom metrics. Parameters: - metric_name (str): The name of the custo

Get Q, K, V tensors from hidden_states, e.g., by applying projections, positional embeddings, KV cache -> Follow the origina

Get Q, K, V tensors from hidden_states, e.g., by applying projections, positional embeddings, KV cache -> Follow the origina

Create a pruned version of a SCROLLS task dataset containing only inputs that are less than `max_tokens` when tokenized by each tokenizer

This function is copy of `random_spans_helper <https://github.com/google-research/text-to-text-transfer-transformer/blob/84f8bcc14b5f2c03de51bd3587609

Compute linear attention with recurrent view -> Assume q.shape is (b, h, 1, d); k and v.shape are (b, h, l, d)

ReLU_ activation function .. _ReLU: https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html

Restore the parameters stored with the `store` method. Useful to validate the model with EMA parameters without affecting the

Find the package root and run the tests for the given tasks

Sample from top-k logits. Arguments: logits: Tensor of shape (batch_size, vocab_size)

Draw `n` samples from our fewshot docs. This method should be overridden by subclasses.

Save model/training states as a checkpoint file through state-dump and file-write. Args: checkpoint: dict containing model and tra

Argument: state: (batch, dim, dstate) x: (batch, dim) dt: (batch, dim) A: (dim, dstate) B: (batch, dstate

Convert the state_dict of a standard GPT model to the state_dict of a GPT model with tensor parallel. This function modifies state_dict in pl

Parses something like args1=val1,arg2=val2 Into a dictionary

tensor: (max_len, d_model)

tensor: (max_len, d_model)

Splits a string on whitespaces and records the indices of each in the original string. @:return generator((word, (start_idx, end_idx)), ...)

Squared ReLU activation, as proposed in the Primer_ paper. .. _Primer: https://arxiv.org/abs/2109.08668