All settings that can't be determined at the time of instantiation need to be passed through here before any dataloaders can be accessed.

Convert a Hugging Face Transformers checkpoint into a LitGPT compatible checkpoint. Arguments: checkpoint_dir: Where to save the dow

Converts lit checkpoint to pytorch model and safetensors Args: checkpoint_name: Filename of the checkpoint directory: Direct

Convert a checkpoint after pretraining. The pretrained checkpoint contains optimizer states and several other metadata that are not needed after

residual_in_fp32 only has an effect if residual is None. Otherwise residual dtype is residual.dtype.

residual_in_fp32 only has an effect if residual is None. Otherwise residual dtype is residual.dtype.

Encodes data using Mistral and Llama tokenizers.

Encode the Capybara dataset by joining 'input' and 'output' fields of each message in the conversation.

Encode datasets with conversation format by joining 'value' fields of each message.

Encode the Llama Instruct dataset by tokenizing the 'text' field.

Encode datasets with message format by joining 'content' fields of each message.

Encode the Meta Math dataset by combining 'query' and 'response' fields.

Measures estimated FLOPs for MFU. Refs: * https://ar5iv.labs.arxiv.org/html/2205.05198#A1 * https://ar5iv.labs.arxiv.org/html/220

x: (batch_size, seqlen, nheads, headdim) cos, sin: (seqlen, rotary_dim / 2) interleaved: if True, rotate pairs of

logits: (batch, vocab_size) labels: (batch,) If process_group is not None, we're doing Tensor Parallel: each process is respo

Do the forward pass. If LoRA's weights are merged with pretrained ones then it's a simple matrix multiplication. If not, then multipl

Args: prompt_template: The prompt template you want to use.

Processes and returns the dataset.

Args: prompt_template: The prompt template you want to use.

Args: split: The dataset split that you want to get. Ultrafeedback has `train` and `test`. prompt_template: The promp

Converts pytorch bins to safetensors. Args: model: Path to the model dir delete: Delete pytorch files after conversion

Sets `requires_grad=False` for all non-adapter weights.

Sets requires_grad=False for all non-adapter weights

Freeze all modules except LoRA's and depending on 'bias' value unfreezes bias weights. Args: model: model with LoRA layers bias: