github.com/SmartFlowAI/TheGodOfCookery @main

中文 | English

🔍 Explore our models:

📍Table of Contents

• 📍Table of Contents
• 📖 Project Introduction
• 🗺️ Technical Architecture
• 1. Overall Technical Architecture
• 2. Application Workflow
• ✨ Technical Report
• 📆 Update Notes
• 🛠️ Usage Guide
• 1. Data Set Preparation
• 2. Installation
• 3. Training
• 4. Dialogue
• 5. Demonstration
• 6. Model Addresses
• 7. Practice Documentation
• 8. Demo Video
• 📋 Project Code Structure
• ☕ Project Members (listed in no particular order)
• 💖 Special Thanks
• License
• Star History

📖 Project Introduction

​ This project, named "The God of Cookery," is inspired by the renowned movie of the same name starring the comedic master, Stephen Chow. The project's goal is to provide cooking advice and recipe recommendations through artificial intelligence technology, helping users to enhance their cooking skills and reduce the barriers to cooking, thereby realizing the movie's message: "With heart, anyone can become a god of cookery." ​ The core concept of this application is based on the InternLM dialogue model, which has been fine-tuned using the XiaChuFang Recipe Corpus, consisting of 1,520,327 Chinese recipes. The model is hosted on ModelScope, and the application is deployed on OpenXLab. Special thanks to the Moda Community for providing free space for model hosting and to OpenXLab for offering the deployment environment and GPU resources. ​ Please note that the answers provided by this application are intended for reference only and should not be considered as actual steps for recipe preparation. Due to the "hallucination" characteristics of large-scale models, some recipes might cause psychological or physiological effects. Users are advised not to take these recipes out of context.

🗺️ Technical Architecture

1. Overall Technical Architecture

The project primarily relies on the open-source model from the Shanghai AI Lab, known as internlm-chat-7b, which includes both first and second generations. We fine-tuned this model on the XiaChuFang Recipe Corpus, which consists of 1,520,327 Chinese recipes. This tuning was facilitated by Xtuner with LoRA fine-tuning, resulting in the creation of the shishen2_full model. Post-tuning, the model was integrated with a vector database into Langchain, achieving an enhanced retrieval effect through RAG (Retrieval-Augmented Generation). It supports multimodal (voice, text, image) question-answering dialogues. The frontend interaction with users is implemented using Streamlit.

2. Application Workflow

Upon receiving a request from a user, the application loads the models (voice model, text-to-image model, fine-tuned dialogue model) and processes the user's text or voice input. If the RAG switch is not activated, it directly calls the fine-tuned dialogue model to generate a reply, formats the result, and uses the stable diffusion model to generate an image, finally returning the result to the user. If the RAG switch is activated, it uses Langchain to search the vector database, inputs the search results into the fine-tuned dialogue model to generate a reply, formats the result, and calls the stable diffusion model to generate an image, ultimately returning the result to the user.

✨ Technical Report

Access the technical report and explanatory videos through the following links:

1. Technical Report

2. Explanatory Video

Section Name	Document Author	Technical Lead
General Overview	zzd2001, chg001, zhanghui-china	zhanghui-china
Voice Recognition	zzd001	sole fish
Text-to-Image	Fang Yuliang	Fang Yuliang
RAG	zzd2001	Charles, Yue Zhengmeng
Model Fine-Tuning	zzd2001	chg001, zzd2001, zhanghui-china
Web UI	Fang Yuliang	Fang Yuliang

📆 Update Notes

• Coming Soon...

• [ ] RAG system based on llama-index and HyQE

• [ ] Speech output

• [ ] Support of other LLMs

• [2024.4.21] HyQE RAG system with LangChain proposed by team member @Yue Zhengmeng merged to main branch

• [2024.3.20] Updated README

• [2024.3.19] Integrated documentation into the docs directory

• [2024.3.9] Based on the RAG module (faiss) by team member @Yue Zhengmeng , integrated the text2image branch, released the fourth phase of the second-generation application based on OpenXLab A100 Click to try it out and OpenXLab A10 application Click to try it out

• [2024.3.4] Added English README

• [2024.3.3] Based on the paraformer voice input module by team member @sole fish, integrated the text2image branch, released the third phase of the second-generation application based on OpenXLab A100 ~~Click to try it out(Link deprecated)~~

• [2024.2.24] Based on the RAG module (Chroma) by team member @Charles, integrated the text2image branch, released the second phase of the second-generation application based on OpenXLab A100 ~~Click to try it out(Link deprecated)~~

• [2024.2.22] Based on the text-to-image module by team member @Fang Yuliang and the whisper voice input module by @sole fish, integrated the text2image branch, released the first phase of the second-generation application(InternLM2-chat-7B as the base model) based on OpenXLab A100 ~~Click to try it out(Link deprecated)~~

• [2024.1.30] Released the model and APP finetuned on the whole 1.5 million recipe based on InternLM-chat-7B (Using InternStudio+A100 1/4X2 40G memory for fine-tuning, from 1.25 15:46 to 1.30 12:25, fine-tuning duration was 4 days 20 hours 39 minutes) by team member @zhanghui-china

• [2024.1.28] Released the model and APP finetuned on a slice of 1.5 million recipe based on InternLM-chat-7B (Using WSL+Ubuntu22.04+RTX4090 24G memory for fine-tuning, from 1.26 18:40 to 1.28 13:46, fine-tuning duration was 1 day 19 hours 6 minutes) by team member @zhanghui-china

🛠️ Usage Guide

1. Data Set Preparation

Download the 1.5 million XiaChuFang fine-tuning dataset: Download Link (password: 8489)

2. Installation

• Set up a Python virtual environment:

conda create -n cook python=3.10 -y
conda activate cook

• Clone the repository:

git clone https://github.com/SmartFlowAI/TheGodOfCookery.git
cd ./TheGodOfCookery

• Install PyTorch and other dependencies:

conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
pip install -r requirements.txt

• To resolve the portaudio compile error when try to run pip install. the error message may like the following:

Building wheels for collected packages: pyaudio
  Building wheel for pyaudio (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for pyaudio (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-310
      creating build/lib.linux-x86_64-cpython-310/pyaudio
      copying src/pyaudio/__init__.py -> build/lib.linux-x86_64-cpython-310/pyaudio
      running build_ext
      building 'pyaudio._portaudio' extension
      creating build/temp.linux-x86_64-cpython-310
      creating build/temp.linux-x86_64-cpython-310/src
      creating build/temp.linux-x86_64-cpython-310/src/pyaudio
      gcc -pthread -B /root/.conda/envs/cook/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/.conda/envs/cook/include -fPIC -O2 -isystem /root/.conda/envs/cook/include -fPIC -I/usr/local/include -I/usr/include -I/root/.conda/envs/cook/include/python3.10 -c src/pyaudio/device_api.c -o build/temp.linux-x86_64-cpython-310/src/pyaudio/device_api.o
      src/pyaudio/device_api.c:9:10: fatal error: portaudio.h: No such file or directory
          9 | #include "portaudio.h"
            |          ^~~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyaudio
Failed to build pyaudio
ERROR: Could not build wheels for pyaudio, which is required to install pyproject.toml-based projects

The root cause is that portaudio19-dev package is required while it is missing in the system, to install it will help to resolve this error before try the pip install operation again.

sudo apt update
sudo apt install -y portaudio19-dev

Note: Choose the CUDA version according to your own CUDA installation, typically 11.8 or 12.1.

3. Training

• Train the first-generation 7b model using xtuner 0.1.9, fine-tune on internlm-chat-7b.
• Train the second-generation 7b model using xtuner 0.1.13, fine-tune on internlm2-chat-7b.
• Train the second-generation 1.8b model using xtuner 0.1.15.dev0, fine-tune on internlm2-chat-1.8b.

Fine-tuning method:

xtuner train ${YOUR_CONFIG} --deepspeed deepspeed_zero2

--deepspeed indicates using DeepSpeed to optimize the training process. XTuner integrates several strategies, including ZeRO-1, ZeRO-2, and ZeRO-3. If you wish to disable this feature, simply remove this parameter.

Convert the saved .pth model (if using DeepSpeed, this will be a directory) into a LoRA model:

export MKL_SERVICE_FORCE_INTEL=1
xtuner convert pth_to_hf ${YOUR_CONFIG} ${PTH} ${LoRA_PATH}

Merge the LoRA model into the HuggingFace model:

xtuner convert merge ${Base_PATH} ${LoRA_PATH} ${SAVE_PATH}

4. Dialogue

xtuner chat ${SAVE_PATH} [optional arguments]

Arguments:

• --prompt-template: Use 'internlm_chat' for the first-generation model and 'internlm2_chat' for the second-generation model.
• --system: Specify the dialogue system identifier.
• --bits {4,8,None}: Specify the LLM's bit rate. Default is fp16.
• --no-streamer: If you want to remove the streamer.
• --top: For second-generation models, a recommendation of 0.8.
• --temperature: