LongLoRA and LongAlpaca for Long-context LLMs

TABLE OF CONTENTS

1. News
2. Highlights
3. How to contribute
4. Requirements
5. Installation and quick guide
6. LongAlpaca Data
7. Models
8. Training
9. Evaluation
10. Demo
11. Streaming Inference
12. Data Generation via Pdf2Text
13. Examples
14. Citation
15. Acknowledgement
16. License

News

• [x] [2024.1.17] LongLoRA has been accepted by ICLR 2024 as an Oral presentation.
• [x] [2023.11.19] We release a new version of LongAlpaca models, LongAlpaca-7B-16k, LongAlpaca-7B-16k, and LongAlpaca-7B-16k. These models are fine-tuned on a subset LongAlpaca-12k dataset with LongLoRA in SFT, LongAlpaca-16k-length. We evaluate the LongAlpaca-7B-16k model on LongBench and L-Eval benchmarks and results can be found here.
• [x] [2023.11.2] We have updated our LongAlpaca models from alpaca prompting to llama2 prompting, which is consistent to their pre-trained models. Please refer to the inference code with the llama2 prompting.
• [x] [2023.10.23] We support the combination of QLoRA and LongLoRA in the supervised fine-tuning, for further reduction of the GPU memory cost. We release the LoRA weights of a 7B model at LongAlpaca-7B-qlora-weights.
• [x] [2023.10.18] We support StreamingLLM inference on our LongAlpaca models. This increases the context-length of the multi-round dialogue in StreamingLLM.
• [x] [2023.10.8] We release the long instruction-following dataset, LongAlpaca-12k and the corresponding models, LongAlpaca-7B, LongAlpaca-13B, and LongAlpaca-70B.
• (The previous sft models, Llama-2-13b-chat-longlora-32k-sft and Llama-2-70b-chat-longlora-32k-sft, have been deprecated.)
• [x] [2023.10.3] We add support GPTNeoX models. Please refer to this PR for usage. Thanks for @naubull2 for this contribution.
• [x] [2023.9.22] We release all our fine-tuned models, including 70B-32k models, LLaMA2-LongLoRA-70B-32k, LLaMA2-LongLoRA-7B-100k. Welcome to check them out!
• [x] [2023.9.22] We release Paper and this GitHub repo, including training and evaluation code.

LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models [Paper]

Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, Jiaya Jia

Highlights

1. In LongLoRA approach, The proposed shifted short attention is easy to implement, compatible with Flash-Attention, and is not required during inference.
2. We released all our models, including models from 7B to 70B, context length from 8k to 100k, including LLaMA2-LongLoRA-7B-100k, LLaMA2-LongLoRA-13B-64k, and LLaMA2-LongLoRA-70B-32k.
3. We built up a long-context instruction-following dataset, LongAlpaca-12k. We released the corresponding LongAlpaca-7B, LongAlpaca-13B and LongAlpaca-70B models. To our best knowledge, this is the first open-sourced long-context 70B model.

How to Contribute

• Make sure to have git installed.
• Create your own fork of the project.
• Clone the repository on your local machine, using git clone and pasting the url of this project.
• Read both the Requirements and Installation and Quick Guide sections below.
• Commit and push your changes.
• Make a pull request when finished modifying the project.

Usage Requirements

To download and use the pre-trained weights you will need: 1. Hugging Face (HF) account with valid email. Note, the email used for HF must alse be used for the license agreement. 2. Accept the Meta license and acceptable use policy

Installation and Quick Guide

To install and run the application: 1. Fork this repo on github 2. Clone the repository on your local machine, using git clone and pasting the url of this project. 3. Run the following code:

pip install -r requirements.txt
pip install flash-attn --no-build-isolation

1. Use either a Released model or Fine tune a model to fit your preferences.
2. Test your model by chat.
3. Deploy your own demo.

LongAlpaca Data

LongAlpaca-12k contains 9k long QA data that we collected and 3k short QA sampled from the original Alpaca data. This is to avoid the case that the model might degrade at short instruction following. The data we collect contains various types and amounts as the following figure.

Following the original Alpaca format, our Long QA data uses the following prompts for fine-tuning: - instruction: str, describes the task the model should perform. For example, to answer a question after reading a book section or paper. We vary the contents and questions to make instructions diverse. - output: str, the answer to the instruction.

We did not use the input format in the Alpaca format for simplicity.

Models

Models with supervised fine-tuning

Models with context extension via fully fine-tuning

Models with context extension via improved LoRA fine-tuning

Training

Pre-trained weights

We use LLaMA2 models as the pre-trained weights and fine-tune them to long context window sizes. Download based on your choices.

This project also supports GPTNeoX models as the base model architecture. Some candidate pre-trained weights may include GPT-NeoX-20B, Polyglot-ko-12.8B and other variants.

Fine-tuning

``` torchrun --nproc_per_node=8 fine-tune.py \ --model_name_or_path path_to/Llama-2-7b-hf \ --bf16 True \ --output_dir path_to_saving_checkpoints \ --cache_dir path_to_cache \ --model_max_length 8192 \ --use_flash_attn True \ --low_rank_training False \ --num_train_epochs 1 \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 2 \ --gradient_accumulation_steps 8 \ --evaluation_strategy "no" \ --save_strategy "steps" \ --save_steps 1000 \ --save_total_limit 2 \ --learning_rate 2e-5 \ --weight_decay 0.0 \ --warmup_steps 20 \ --lr_scheduler_type "constant_with_warmup" \ --logging_st