🤗 Hugging Face • 🤖 ModelScope • 👾 Wisemodel • 💬 WeChat• 📜Tech Report

Project Introduction

We are pleased to announce the open source release of the Skywork large-scale models. Skywork is a series of large models developed by the Kunlun Group · Skywork team. The models being open sourced this time include the Skywork-13B-Base model, Skywork-13B-Chat model, Skywork-13B-Math model, and Skywork-13B-MM model, as well as quantized versions of each model to support deployment and inference on consumer-grade GPUs.

Our open-source Skywork series models can be used for commercial purposes, but you need to follow our agreement and refrain from engaging in harmful activities. The characteristics of the Skywork open-source project are:：

• Skywork-13B-Base: The model was trained on a high-quality cleaned dataset consisting of 3.2 trillion multilingual data (mainly Chinese and English) and code. It has demonstrated the best performance among models of similar scale in various evaluations and benchmark tests.

• Skywork-13B-Chat: The model has powerful conversational abilities, and we have further enhanced it in the field of creative writing. We have constructed a high-quality dataset of over ten thousand instructions and fine-tuned the model on ten specific creative writing tasks, enabling it to achieve results similar to ChatGPT in these tasks. Additionally, we open-source a benchmark consisting of approximately 500 samples for these 10 creative writing tasks.

• Skywork-13B-Math: This model has undergone specialized training to enhance its mathematical abilities. In the 13B-scale model, the Skywork-13B-Math model ranked 1st in the GSM8K benchmark, and it also performed exceptionally well on the MATH and CMATH benchmarks, placing it among the top-level 13B models.

• Skywork-13B-MM: This is a multimodal model that allows users to utilize image information for tasks like Q&A and dialogue.

• Skywork/Skypile-150B: This dataset is a collection of high-quality data extracted from Chinese web pages through our carefully curated data processing pipeline. The size of this open-source dataset is approximately 600GB, with a total token count of around 150 billion. It is one of the largest publicly available Chinese datasets.

• In addition, we have also disclosed the evaluation methods, data distribution studies, and training infrastructure optimization plans used in training the Skywork-13B model. We hope that these open-source materials can further inspire the community's understanding of large-scale model pre-training and drive the realization of Artificial General Intelligence (AGI).

If you are interested in more training and evaluation details, please refer to our technical report, Skymath paper and SkyworkMM paper.

News and Updates

• 2023.12.7 Our SkyPile-150B dataset is now accessible via huggingface.

• 2023.11.2 We have uploaded the evaluation data we built, MOCK_GSM8K_TEST, and the Chinese domain evaluation data ChineseDomainModelingEval to huggingface. If you need to evaluate LLMs, please download our evaluation dataset.

• 2023.10.31 Our technical report Skywork: A More Open Bilingual Foundation Model is available on arXiv, which includes more detailed evaluation methods, result comparisons, and technical details.

• 2023.10.30 We release the Skywork-13B-Base and Skywork-13B-Math models, as well as quantized versions of each model to support deployment and inference on consumer-grade GPUs. We open-source the Skywork/Skypile-150B dataset. This dataset contains over 150 billion high-quality tokens cleaned from Chinese web pages, making it the largest open-source Chinese dataset currently known.

Table of contents

• ☁️Download URL
• 👨‍💻Model Introduction
• 🏆Model Evaluation
• 📕Quickstart
• 📣Chat Model Output Examples
• 🚀Quantization
• 🛫Fine-tuning
• 🍀Community and Ecosystem
• ⚠️Declaration and License Agreement
• 🤝Contact Us and Citation

Download URL

Download URL of Skywork Models

	HuggingFace Base Model	HuggingFace Quantized Model	ModelScope Base Model	ModelScope Quantized Model	Wisemodel Base Model	Wisemodel Quantized Model
Skywork-13B-Base	🤗 Skywork-13B-Base	🤗 Skywork-13B-Base-8bits	🤖Skywork-13B-Base	🤖 Skywork-13B-Base-8bits	👾Skywork-13B-Base	👾 Skywork-13B-Base-8bits
Skywork-13B-Chat	🤗coming soon	🤗coming soon	🤖coming soon	🤖coming soon	👾coming soon	👾coming soon
Skywork-13B-Math	🤗 Skywork-13B-Math	🤗 Skywork-13B-Math-8bits	🤖 Skywork-13B-Math	🤖 Skywork-13B-Math-8bits	👾Skywork-13B-Math	👾 Skywork-13B-Math-8bits
Skywork-13B-MM	🤗coming soon	-	🤖coming soon	-	👾coming soon	-

Download URL of Skypile

Data	Download URL
Skywork/Skypile-150B	Hugging Face URL

Download of Intermediate Model Checkpoints

We have also open-sourced the Skywork-13B-Base model and provided the model checkpoints trained on 500B, 1T, 1.5T, 2T, 2.5T, 3T and 3.1T tokens for community research into the evolution process of large language model capabilities.

Model	Download URL
Skywork-13B-Base-Intermediate	🤗Skywork-13B-Base-Intermediate
Skywork-13B-Base-3.1T	🤗Skywork-13B-Base-3.1T

Skywork-13B Introduction

Training Data

We have developed a data cleaning pipeline with great care to effectively clean and filter low-quality data and eliminate harmful information from text data. Our Skywork-13B-Base model is trained on a dataset with 3.2T tokens that consists of high-quality Chinese, English, and code data, all of which have been thoroughly cleaned. The English data comprises 52.2% of the dataset, the Chinese data accounts for 39.6%, and the code data makes up 8%. This comprehensive approach ensures optimal performance for both Chinese and English while also maintaining the ability to handle code. | | Category | Percentage | |-------------|------------------|------------| | English | Webpages | 39.8% | | | Books | 3.6% | | | Academic Papers | 3.0% | | | Encyclopedia | 0.5% | | | Miscellany | 2.9% | | Chinese | Webpages | 30.4% | | | Social Media | 5.5% | | | Encyclopedia | 0.8% | | | Miscellany | 3.1% | | Other Lang. | Encyclopedia | 2.4% | | Code | Github | 8.0% |

Model Structure

Compared to the Llama-2-13B model, the Skywork-13B model adopts a relatively thinner and deeper network structure with 52 layers. At the same time, the FFN Dim and Hidden Dim are reduced to 12288 and 4608, respectively, to ensure that the model has a similar number of parameters as the original Llama-2-13B model. Based on our preliminary experimental results, a relatively thinner and deeper network structure can achieve better generalization performance under large batch size training. The detailed comparison between the Skywork-13B and Llama-2-13B models is as follows:

Model Structure	Llama-2-13B	Skywork-13B
Vocab. Size	32,000	65,536
Hidden Dim.	5,120	4,608
FFN Dim.	13,696	12,288
Head Dim.	128	128
Num. Heads	40	36
Num. Layers	40	52
Seq. Len.	4,096	4,096
Positional Embedding	RoPE	RoPE

Tokenizer

We use Byte-Pair Encoding (BPE) to tokenize the data, with a vocabulary size of 65536. Among them, there are 32000 Latin characters and subwords, 8000 Chinese characters and Unicode symbols, 25519 Chinese words, and the remaining 17 are reserved words.

Category	Size
Latin based words & subwords	32000
Chinese characters & Unicode symbols	8000
Chinese words	25519
Reserved symbols	17
Total	65536

Training Methods

In order to make more precise use of data, we adopt a two-stage training method. In the first stage, we use general corpora to train the model's general abilities. In the second stage, we incorporate STEM (Science, Technology, Engineering, Mathematics) related data to further enhance the model's reasoning, mathematical, and problem-solving abilities.

First-stage Pretraining

During the training process, we monitor the changes in model training loss and various abilities. The following figure shows the change curves of important indicators selected during the first stage of pre-training. The first stage of pre-training consists of two consecutive training processes, which are represented by different colors. The model completed in the first stage of pre-training is referred to as Skywork-13B-3.1T-Base.

Second-stage Pretraining

In the second stage of pre-training, STEM-related data is added to the general language corpus for further training. The second stage training involves approximately 130 billion tokens, resulting in a total training of 3.2 T across both stages, and yielding our final Skywork-13B-Base model.

Skypile-150B

Introduction

Skypile-150B is a large dataset specifically designed for pre-training Chinese language models. It is constructed using publicly available web page data from the Chinese internet. The dataset has undergone extensive filtering to remove duplicate and harmful text. Additionally, advanced models like FastText and Bert have been employed to further refine the dataset and eliminate low-quality data.

Language and Data Format

Skypile-150B dataset is a collection of Chinese data. The pages contain processed and cleaned text, in JSONL format. Each line represents a document, parsed using JSON. The text is stored in the "text" field.

Sensitive information and bias

Although it has undergone strict cleaning and filtering, since it is built on a publicly accessible webpage established by Skypile-150B, it may still contain some sensitive information such as email addresses, phone numbers, or IP addresses. Therefore, users need to be careful and perform necessary additional filtering and cleaning before using the data.

License Agreement

The use of data must comply with our License and must not be used for any purpose that poses a threat to national and social security or violates the law.

Model Evaluation

Documentation Perplexity Evaluation

The main goal of training a language model is to improve the accuracy of predicting the next word. With this in mind, we believe that evaluating the ability of a language model to generate articles in different domains is a crucial way to assess the performance of large-scale models. During model training, the likelihood of predicting the next word is typically measured using the Cross Entropy loss function. The overall loss function is calculated as the average of the losses when predicting the correct word at each position, which can be represented as:

loss = -\sum^{n}_{i=1} log(p_i) / n = -log( \prod_{i=1}^n p_i) / n

Where $n$ is the length of