Measures real FLOPs for HFU

Merges the LoRA weights into the full-rank weights (W = W + delta_W).

Merge LoRA weights into the full-rank weights to speed up inference.

This defines how the file will be accessed. NOTE: This does not mandate a return type, as different files will return different types

Calls `reset_parameters` on the module and all its submodules.

Overridden to merge CSV by the step number.

Finetune a model. Arguments: checkpoint_dir: The path to the base model's checkpoint directory to load for finetuning. out_dir: D

Sets up the environment and configurations for training a model. Args: data_dir (Optional[Path]): Directory containing the training

Test that the packed dataset is correctly formatted with bos and eos tokens, even when multiple sequences are packed into it.

Test that the tokenizer correctly encodes a single string with bos and eos tokens.