English ｜ <a href="https://github.com/LlamaChinese/Llama-Chinese/raw/main/README.md">中文</a>

Llama-Chinese

The Best Chinese Llama Large Language Model

🤗 Hugging Face • 🤖 ModelScope • ✡️ WiseModel

Online(Including Llama2, Llama3): llama.family

Open-source Chinese Pre-trained LLM Atom based on Llama2

🗂️ Content Guide

• 📌 Chinese Llama Community
• 🔥 Community Introduction: Chinese Llama Community
• 📢 Community Announcements
• 🤗 LLM Model
  • 🤗 Pre-trained Chinese Model Atom based on Llama2
  • 🤗 Meta Official Llama2 Model
  • 🤗 Fine-tuned Chinese Models based on Llama2

• 🌟 Community Source

  • GPU Source
  • Data Source
  • Discussion
  • Product

• 📌 How to use Llama Model?

• Setup of Llama3
• Setup of Llama2
  • Simple Setup
  • Simple Setup-Anaconda
  • Simple Setup-Docker
  • Simple Setup-llama.cpp
  • Simple Setup-gradio
  • Simple Setup-API
  • 🤖 Model Pretraining
  • 💡 Model Fine-tuning
  • Step1: Environment Setup
  • Step2: Data Preparation
  • Step3: Fine-tuning Scripts
    • LoRA Fine-tuning
    • Full-parameter Fine-tuning
  • Step4: Load Fine-tuned Model
    • LoRA Fine-tuning
    • Full-parameter Fine-tuning
  • 🍄 Model Quantization
  • 🚀 Inference Acceleration
  • TensorRT-LLM
  • vLLM
  • JittorLLMs
  • lmdeploy
  • 💪 Extension Capabilities
  • LangChain
• 🥇 Model Evaluation

• 📖 Learning Resources

  • Llama3
  • Llama2
  • Meta Official Introduction to Llama2
  • Llama-related Papers
  • Llama2 Evaluation Results

• 📌 Others

• 🎉 Acknowledgments
• 🤔 Issue Feedback

📌 Chinese Llama Community

🔥 Community Introduction: Chinese Llama Community

Welcome to the Chinese Llama Community! We are a technical community dedicated to optimizing and building on top of the Llama model for Chinese applications. *Based on large-scale Chinese data, we start pre-training and continuously upgrade the Llama2 model for Chinese capabilities*. We warmly welcome developers and researchers passionate about LLM models to join our community.

Why Choose the Chinese Llama Community?

🚀 Support from a Team of Senior Engineers: The community has a team of dedicated NLP senior engineers who provide strong technical support and rich experience to guide and assist you.

🎯 Chinese Optimization: We focus on optimizing Llama2 for Chinese processing, exploring the best practices for Chinese to enhance its performance and adaptability.

💡 Innovative Exchange: Our community includes a creative and experienced team of members who organize regular online events, technical discussions, and experience sharing to promote innovative exchanges.

🌐 Global Connectivity: We welcome developers from around the world to join the community, creating an open and diverse platform for learning and communication.

🤝 Open Sharing: We encourage community members to open-source and share code and models, promoting collaborative win-win efforts and advancing the development of Chinese NLP technology.

Community Activities

🗓️ Online Lectures: Inviting industry experts to conduct online lectures, sharing the latest technology and applications of Llama2 in the Chinese NLP field, and discussing cutting-edge research results.

💻 Project Showcase: Members can showcase their project achievements in Llama2 Chinese optimization, receive feedback and suggestions, and promote project collaboration.

📚 Learning Resources: The community maintains a rich library of learning materials, including tutorials, documentation, and paper interpretations, providing comprehensive learning support to members.

📝 Paper Interpretation: Community members collectively interpret the latest research papers related to Llama2, delving into advanced algorithms and methods.

🎉 Themed Events: Regularly organize various themed events, including challenges, hackathons, and technical salons, allowing community members to exchange and learn in a relaxed and enjoyable atmosphere.

🌟 Reward Program: We have established a reward program to honor and reward members who actively participate and contribute outstanding work to the community, motivating more outstanding talents to join.

📈 Technical Consultation: We provide technical consulting services to answer your questions and help you overcome challenges in the development and optimization of Llama2.

🚀 Project Collaboration: Encourage collaboration between members on projects to explore the potential of Llama2 in practical applications and create innovative solutions.

Join Us Now!

📚 Vision: Whether you are a professional developer or researcher with experience in Llama2 or a newcomer interested in optimizing Llama2 for Chinese, we eagerly look forward to your joining. In the Chinese Llama Community, you will have the opportunity to exchange ideas with top talents in the industry, work together to advance Chinese NLP technology, and create a brighter technological future!

🔗 Friendly Reminder: This community is a platform for professional technical exchange. We earnestly hope that like-minded developers and researchers join us. Please adhere to the community guidelines, maintain a positive learning atmosphere, and any content and advertisements unrelated to Llama2 will be removed. Thank you for your understanding and support!

📢 Community Announcements

【Latest】October 8, 2023: Added the inference acceleration feature for JittorLLMs from Tsinghua University JittorLLMs!

【Latest】September 12, 2023: Updated pre-training versions Atom-7B and dialogue version Atom-7B-Chat model parameters. The latest Chinese pre-training data size is 100 billion tokens, and the training progress can be viewed at llama.family!

【Latest】September 2, 2023: Added pre-training code and full-parameter fine-tuning code!

【Latest】August 28, 2023: Released the open-source large model Atom-7B based on Llama2 for Chinese pre-training and will continue to be updated. Details can be found in the community article!

【Latest】August 26, 2023: Provided FastAPI interface setup script!

【Latest】August 26, 2023: Provided a script to convert Meta official model parameters to a format compatible with Hugging Face Format Conversion Script!

【Latest】August 26, 2023: Added Code Llama model!

• August 15, 2023: Added PEFT load fine-tuning model parameters code example!

• August 14, 2023: Launched the large model data sharing training platform, allowing everyone to contribute to large model training, even without computing resources. The data contributed by each community member will determine the future capabilities of the model!

• August 3, 2023: Added GPU inference acceleration support for FasterTransformer and vLLM!

• July 31, 2023: 【Major】The first truly meaningful Llama2 Chinese large model is released! Details can be found in the community article

• July 28, 2023: Deployed a Q&A interface through Docker!

• July 27, 2023: Added LangChain support!

• July 26, 2023: Released a 4-bit quantized compressed version of the Llama2-13B Chinese fine-tuning parameters!

• July 25, 2023: The community's WeChat public account "Llama Chinese Community" is now live. Feel free to follow for the latest updates and dynamics!

• July 24, 2023: FlagAlpha added Llama2-13B Chinese fine-tuned parameters!

• July 24, 2023: llama.family added Llama2-70B online experience!

• July 23, 2023: Released Llama2-13B Chinese fine-tuned parameters to the Hugging Face repository FlagAlpha!

• July 22, 2023: Llama2 online experience link llama.family is live, including both Meta original and Chinese fine-tuned versions!

• July 21, 2023: Evaluated the Chinese Q&A capability of the Meta original Llama2 Chat model Model Evaluation!

• July 21, 2023: Added the Hugging Face version download link for Llama2 models in China!

• July 20, 2023: Added Feishu Knowledge Base Documentation, welcome everyone to contribute!

• July 20, 2023: Chinese Llama2 latest download links are live!

• July 19, 2023: Officially launched the Llama2 Chinese community, stay tuned for real-time updates!

• July 19, 2023: Chinese Llama2 latest download links are in progress, stay tuned!

• July 19, 2023: Launched the Llama2 Chinese community, welcome everyone to join!

🤗 Models Downloads

🔵 Atom Models

The Atom models, created jointly by the Chinese Llama Community and AtomEcho, rank in the top ten of the Chinese Large Language Model Evaluation List C-Eval (submission on August 21).

Category	Model Name	🤗Model Loading Name	Download Link
Pretrained	Atom-7B	FlagAlpha/Atom-7B	HuggingFace | ModelScope | WiseModel
Chat	Atom-7B-Chat	FlagAlpha/Atom-7B-Chat	HuggingFace | ModelScope | WiseModel

The Atom series includes Atom-1B, Atom-7B and Atom-13B, with continuous optimization of Chinese language proficiency based on Llama2. Atom-7B and Atom-7B-Chat are fully open source and available for commercial use. You can obtain the models on the Hugging Face repository. Details are available in Atom-7B Download.

Atom models have the following optimizations for Chinese:

Large-scale Chinese Data Pretraining

Atom models are continually pretrained using a large amount of Chinese data, including encyclopedias, books, blogs, news, announcements, novels, financial data, legal data, medical data, code data, professional paper data, and Chinese natural language processing competition datasets. See 📝 Data Sources for details.

The massive data is filtered, scored, and deduplicated, resulting in high-quality Chinese data exceeding 1T tokens, continuously added to the training iterations.

More Efficient Chinese Vocabulary

To improve the efficiency of Chinese text processing, we optimized the vocabulary of the Llama2 model. First, based on several hundred gigabytes of Chinese text, we expanded the word library to 65,000 words on the basis of the model's vocabulary. Our improvements increased the Chinese encoding/decoding speed by about 350% according to tests. Additionally, we expanded the coverage of the Chinese character set, including all emoji symbols 😊. This makes generating articles with emoji symbols more efficient.

Adaptive Context Expansion

Atom large models support a default context of 4K. Through position interpolation (PI) and Neural Tangent Kernel (NTK) methods, the context length can be expanded to 32K after fine-tuning.

📝 Chinese Data

We optimized the Chinese capabilities of Llama2 using the follo