NExT-GPT: Any-to-Any Multimodal LLM

Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. (Correspondence )

ICML 2024, Oral Paper

NExT++ Research Center, School of Computing, National University of Singapore

---

This repository hosts the code, data and model weight of NExT-GPT, the first end-to-end MM-LLM that perceives input and generates output in arbitrary combinations (any-to-any) of text, image, video, and audio and beyond.

Noted: we wrap the former old codebase into the NExT-GPT-Lagacy. Please refer to this new codebase for all training and tuning procedures.

---

🎉 News

• [x] [2023.09.15] 🚀🚀 Release the code of NExT-GPT in version 7b_tiva_v0.
• [x] [2023.09.27] 🔨🧩 Added modality-blended batch sampler.
• [x] [2023.10.01] 📢📢 Release the T2M instruction dataset.
• [x] [2023.10.04] 👏👏 Release the checkpoint of NExT-GPT in version 7b_tiva_v0 .
• [x] [2023.10.15] 🔨🚀 Update of NExT-GPT in version 7b_tiva_v0 .
• [x] [2024.10.07] 👏👏 Release the data and the corresponding construction methods, please refer DATA_README.md for more details.

👉 TODO

• [ ] Updating NExT-GPT in more types&sizes of LLMs.
• [ ] Empowering NExT-GPT with more modalities of inputs&outputs.
• [ ] ...

---

Example Demos

Here we showcase examples generated from NExT-GPT. For more examples, kindly visit the webpage, or the online live demo.

https://github.com/NExT-GPT/NExT-GPT/assets/18722770/0c2b3d88-a533-4899-ab44-65580fe54538

https://github.com/NExT-GPT/NExT-GPT/assets/18722770/eb1319a6-38aa-4546-a96e-163207e7de93

https://github.com/NExT-GPT/NExT-GPT/assets/18722770/36bec0ad-9bad-4bcf-bc37-92b028f1bc6a

Brief Introduction

NExt-GPT is built on top of existing pre-trained LLM, multimodal encoder and SoTA diffusion models, with sufficient end-to-end instruction tuning.

• Multimodal Encoding Stage. Leveraging established encoders to encode inputs in various modalities, where these representations are projected into language-like representations comprehensible to the LLM through a projection layer.
• LLM Understanding and Reasoning Stage. Harnessing an existing open-sourced LLM as the core to process input information for semantic understanding and reasoning. The LLM not only directly generates text tokens but also produces unique “modality signal” tokens that serve as instructions to dictate the decoding layers whether & what modal content to output correspondingly.
• Multimodal Generation Stage. Receiving the multimodal signals with specific instructions from LLM (if any), the Transformer-based output projection layers map the signal token representations into the ones that are understandable to following multimodal decoders.

For more technical details, kindly refer to the paper.

---

Getting Started

Table of Contents:

• 1. Code Structure
• 2. Environment Preparation
• 3. Training/Adapting NExt-GPT on Your Own
• 3.1. Preparing Pre-trained Checkpoint
• 3.2. Preparing Dataset
• 3.3. Precomputing Embeddings
• 3.4. Training NExT-GPT
• 4. Running NExT-GPT System
• 4.1. Preparing checkpoints
• 4.2. Deploying Demo System
• 5. Fine-tuning your own System
• 5.1. Dataset
• 5.2. Model Framework
• 5.3. Fine-tuning

---

1. Code Structure

.
|-- NExT-GPT-Lagacy       # the previous version of the model
|-- assets
|-- checkpoints           # save the pretraining and tuning checkpoints
|-- data  
|   |-- IT_data
|   |   |-- MosIT_data
|   |   |-- T+X-T_data    # text+[image/audio/video] to text instruction data
|   |   `-- T-T+X_data    # synthesized text to text+[image/audio/video] instruction data
|   |-- T_X_pair_data     # text-autio pairs data
|   |   |-- audiocap
|   |   |-- cc3m
|   |   `-- webvid
|   |-- embed 
|   `-- prepare_data.py
|-- figures
|-- merge_lora_weights.py
|-- nextgpt
|   |-- __init__.py
|   |-- constants.py
|   |-- conversation.py
|   |-- dataset
|   |   |-- __init__.py
|   |   |-- audio_processor.py
|   |   |-- base_dataset.py
|   |   |-- catalog.py
|   |   |-- concat_dataset.py
|   |   |-- dataset_utils.py
|   |   `-- sampler.py
|   |-- mm_utils.py
|   |-- model
|   |   |-- __init__.py
|   |   |-- apply_delta.py
|   |   |-- builder.py
|   |   |-- consolidate.py
|   |   |-- language_model
|   |   |-- make_delta.py
|   |   |-- multimodal_decoder
|   |   |-- multimodal_encoder
|   |   |-- multimodal_projector
|   |   |-- nextgpt_arch.py
|   |   `-- utils.py
|   `-- utils.py
|-- scripts
|   |-- finetune.sh
|   |-- pretrain_dec.sh
|   |-- pretrain_enc.sh
|   |-- zero2.json
|   |-- zero3.json
|   `-- zero3_offload.json
|-- LICENSE.md
|-- README.md
|-- nextgpt_trainer.py
|-- predict.py
|-- preprocess_embeddings.py
|-- requirements.txt
|-- train.py
|-- train_mem.py
`-- training_utils.py

2. Environment Preparation [Back to Top]

Please first clone the repo and install the required environment, which can be done by running the following commands:

conda env create -n nextgpt python=3.8

conda activate nextgpt

# CUDA 12.1
conda install pytorch==2.1.2 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia

git clone https://github.com/NExT-GPT/NExT-GPT.git
cd NExT-GPT

pip install -r requirements.txt

3. Training/Adapting NExt-GPT on Your Own

3.1. Preparing Pre-trained Checkpoint [Back to Top]

NExT-GPT is trained based on following excellent existing models. Please follow the instructions to prepare the checkpoints.

• ImageBind is the unified image/video/audio encoder. The pre-trained checkpoint can be downloaded from here with version huge. Afterward, put the imagebind_huge.pth file at [.pretrain_ckpt/imagebind].
• Vicuna: prepare the pretrained vicuna from [here]. Then put the pre-trained model at [./pretrain_ckpt/vicuna-7b-v1.5/].
• Image Diffusion is used to generate images. NExT-GPT uses Stable Diffusion with version v2. (will be automatically downloaded)
• Audio Diffusion for producing audio content. NExT-GPT employs AudioLDM with version l-full. (will be automatically downloaded)
• Video Diffusion for the video generation. We employ ZeroScope with version v2_576w. (will be automatically downloaded)

3.2. Preparing Dataset [Back to Top]

Please download the following datasets used for model training:

A) T-X pairs data - CC3M of text-image pairs, please follow this instruction [here]. Then put the data at [./data/T-X_pair_data/cc3m]. - WebVid of text-video pairs, see the [instruction]. The file should be saved at [./data/T-X_pair_data/webvid]. - AudioCap of text-audio pairs, see the [instruction]. Save the data in [./data/T-X_pair_data/audiocap].

B) Instruction data - T+X-T - LLaVA of the visual instruction data, download it from here, and then put it at [./data/IT_data/T+X-T_data/llava]. - Alpaca of the textual instruction data, download it from here, and then put it at [./data/IT_data/T+X-T_data/alpaca/]. - VideoChat, download the video instruction data here, and then put it at [./data/IT_data/T+X-T_data/videochat/].

Side note：After downloading dataset, please run `prepare_data.py` to preprocess the dataset.

• T-X+T (T2M)

  • The T-X+T instruction datasets (T2M) are saved at [./data/IT_data/T-T+X_data].

• MosIT

  • Download the file from here, put them in [./data/IT_data/MosIT_data/]. (We are in the process of finalizing the data and handling the copyright issue.)

3.3. Precomputing Embeddings [Back to Top]

In decoding-side alignment training, we minimize the distance between the representation of signal tokens and captions. To save costs of time and memory, we precompute the text embeddings for image, audio and video captions using the text encoder within the respective diffusion models.

Please run this command before the following training of NExT-GPT, where the produced embedding file will be saved at [./data/embed].

cd ./code/
python preprocess_embeddings.py ../data/T-X_pair_data/cc3m/cc3m_generation.json image ../data/embed/ stabilityai/stable-diffusion-2

Note of arguments: - args[1]: path of caption file; - args[2]: modality, which can be image, video, and audio; - args[3]: saving path of embedding file; - args[4]: corresponding pre-trained diffusion model name.

3.4. Training NExT-GPT [Back to Top]

First of all, please refer to the base configuration file [training_utils.py] for the basic system setting of overall modules, and dataset configuration nextgpt/dataset/catalog.py. The whole NExT-GPT training involves 3 steps:

• Step-1: Encoding-side LLM-centric Multimodal Alignment. This stage trains the input projection layer while freezing the ImageBind, LLM, output projection layer. angular2html # Encoding-side LLM-centric Multimodal Alignment bash scripts/pretrain_enc.sh

• Step-2: Decoding-side Instruction-following Alignment. This stage trains the output projection layers while freezing the ImageBind, LLM, input projection layers. angular2html # Encoding-side LLM-centric Multimodal Alignment bash scripts/pretrain_enc.sh

• Step-3: Instruction Tuning. This stage instruction-tune 1) the LLM via LoRA, 2) input projection layer and 3) output projection layer on the instruction dataset. angular2html # Encoding-side LLM-centric Multimodal Alignment bash scripts/pretrain_enc.sh

4. Running NExT-GPT System [Back to Top]

4.1. Preparing Checkpoints

First, loading the pre-trained NExT-GPT system. - Step-1: load Frozen parameters. Please refer to 3.1 Preparing Pre-trained Checkpoint.

• Step-2: load Tunable parameters. Please put the NExT-GPT system at ./checkpoints/nextgpt-v1.5-7b. You may either 1) use the params trained yourselves, or 2) download our checkpoints from Huggingface.

4.2. Run the Prediction

Upon completion of the checkpoint loading, you can run the prediction via:

python predict.py

---

5. Fine-tuning Your Own System [Back to Top]

5.1. Dataset

You can define your own dataset, please refer to the base_dataset.py, and then add the dataset catalog in catalog.py, including the target and parameters.

5.2. Model Framework

• Multimodal Encoder: You can leverage your own multimodal encoder in multimodal encoder directory, and add corresponding code in the [builder.py](nextgpt/model/mult