MatFormer: Nested Transformer for Elastic Inference

This repository provides a public reproduction and open-source implementation of the MatFormer's language modeling experiments (MatLM). It includes the essential building blocks and code required to reproduce the results presented in the paper.

Features

• Simplified implementation of MatFormer for language modeling tasks.
• Open-source release for community use and further research.
• Reproducibility: Includes key components to replicate the experiments.

Running LM Pre-training Jobs

To run the training script, execute:

python3 train.py