Language: 简体中文 English

This English readme will not update first time, please read 简体中文 for latest info。

I am not good at English. If there are any errors, please point them out.

Diffusion-SVC

This repository is a separate storage for the diffusion part of the DDSP-SVC repository. It can be trained and inferred independently.

---

Recent update: The use of the naive model and the shallow Diffusion model of the repository can achieve better results than the simple Diffusion model with extremely low training costs, which is strongly recommended.However, the Naive model has weak generalization ability and may have too small f0 range on small datasets. At this point, the Naive model cannot train too many steps for fine-tuning (which will degrade the base model), and the front stage can also be considered to be replaced with an infinite range ddsp model.

Samples and introductions can be found in [Introduction Video(Not done yet)]

0. Introduction

Diffusion-SVC is a separate storage for the diffusion part of the DDSP-SVC repository. It can be trained and inferred independently.

Compared with the well-known Diff-SVC, this project consumes much less graphic memory, has faster training and inference speed, and is specially optimized for shallow diffusion and real-time use. It can perform real-time inference on a powerful GPU.By combining with the Naive model of this project for shallow diffusion, even weaker GPUs can generate high-quality audio in real-time.

If the quality of the training data and the input source are both very high, Diffusion-SVC may have the best conversion effect.

This project can easily cascade shallow diffusion after other Acoustic model to improve the final output effect or reduce the performance occupancy. For example, cascading Diffusion SVC after the Naive model of this project or DDSP SVC can further reduce the required diffusion steps and obtain high-quality output.

In addition, this project can also train the number of noise reduction steps required for shallow diffusion without training the complete noise reduction process starting from Gaussian noise, which can improve the training speed and quality. See the following for more information.

Disclaimer: Please ensure to only use legally obtained authorized data to train the Diffusion-SVC model, and do not use these models and any audio synthesized by them for illegal purposes. The author of this library is not responsible for any infringements, scams, and other illegal acts caused by the use of these model checkpoints and audio.

1. Install dependencies

1. Install PyTorch: We recommend first downloading PyTorch from the PyTorch official website, then run:

pip install -r requirements.txt 

2. Configure the requirement model

• (Required) Download the pretrained ContentVec encoder and place it in the pretrain folder. Cropped ContentVec images has the same effect, but only 190MB.
• Note: You can also use other feature extractors, but ContentVec is still highly recommended. All supported feature extractors can be found in the Units_Encoder class in tools/tools.py.
• (Required) Download the pretrained vocoder from the DiffSinger community vocoder project and unzip it to the pretrain/ folder.
• Note: You should download the compressed file with nsf_hifigan in its name, not nsf_hifigan_finetune.
• ~~If you need to use the voiceprint model, you need to set use_speaker_encoder in the configuration file to true, and download the pretrained voiceprint model from here. This model comes from mozilla/TTS.~~

3. Preprocessing

1. Setting up Training and Validation Datasets

1.1 Manual Configuration:

Place all training set data (.wav format audio clips) in the data/train/audio folder, or in a directory specified in the configuration file such as xxxx/yyyy/audio.

Place all validation set data (.wav format audio clips) in the data/val/audio folder, or in a directory specified in the configuration file such as aaaa/bbbb/audio.

~~1.2 Random Selection by Program (Not Implemented):~~

~~Run python draw.py. The program will help you select validation set data (you can adjust parameters such as the number of files to be extracted in draw.py).~~

1.3 Folder Structure Directory Display:

Note: Speaker IDs must start from 1, not 0; if there is only one speaker, this speaker's ID must be 1. - Directory Structure:

data
├─ train
│    ├─ audio
│    │    ├─ 1
│    │    │   ├─ aaa.wav
│    │    │   ├─ bbb.wav
│    │    │   └─ ....wav
│    │    ├─ 2
│    │    │   ├─ ccc.wav
│    │    │   ├─ ddd.wav
│    │    │   └─ ....wav
│    │    └─ ...
|
├─ val
|    ├─ audio
│    │    ├─ 1
│    │    │   ├─ eee.wav
│    │    │   ├─ fff.wav
│    │    │   └─ ....wav
│    │    ├─ 2
│    │    │   ├─ ggg.wav
│    │    │   ├─ hhh.wav
│    │    │   └─ ....wav
│    │    └─ ...

2. Start Preprocessing

python preprocess.py -c configs/config.yaml

You can modify the configuration file configs/config.yaml before preprocessing.

3. Notes:

1. Please ensure that the sampling rate of all audio clips matches the sampling rate specified in the yaml configuration file! (If pre processing such as resampling is required, it is recommended to use fap)

2. Cutting long audio into smaller clips can speed up training, but the duration of all audio clips should not be less than 2 seconds. If there are too many audio clips, you will need more memory. Setting the cache_all_data option in the configuration file to false can solve this problem.

3. It is suggested that the total number of audio clips in the validation set be about 10, don't put too many, otherwise the validation process will be slow.

4. If your dataset quality is not very high, please set the 'f0_extractor' in the configuration file to 'crepe'. The crepe algorithm has the best noise resistance, but it will significantly increase the time required for data preprocessing.

5. The ‘n_spk’ parameter in the configuration file will control whether to train a multi-speaker model. If you want to train a multi-speaker model, to number the speakers, all audio folder names must be integers no larger than ‘n_spk’.

4. Training

1. Training without pre-trained model:

python train.py -c configs/config.yaml

2. Pre-trained Models:

• We strongly recommend fine-tuning with pre-trained models, which is easier and much more time-saving than training from scratch, and can achieve a higher limit than small datasets.

• Please note, fine-tuning on a base model requires using the same encoder as the base model, such as ContentVec, the same applies to other encoders (like voiceprint), and the model's network size and other parameters should be the same.

---

！！！！！！Recommend training Shallow Diffusion Model and Naive Model！！！！！！

The combination of shallow Diffusion model that only train k_step_max depth and Naive model may have higher quality and faster training speed than pure full diffusion model. But the Naive model may have f0 range issues.

---

2.1 Pre training Diffusion model which training full depth

Units Encoder	Network size	Datasets	Model
contentvec768l12(Recommend)	512*20	VCTK

m4singer | HuggingFace | | hubertsoft | 512*20 | VCTK

m4singer | HuggingFace | | whisper-ppg(only can use with sovits) | 512*20 | VCTK

m4singer

opencpop

kiritan | HuggingFace |

Here is an additional special pre-trained model using the contentvec768l12 encoder, the dataset is m4singer/opencpop/vctk. It is not recommended to use this and there's no guarantee it won't cause problems: Download.

2.2 Diffusion pre training model that only trains k_step_max depth

Units Encoder	Network size	k_step_max	Datasets	Diffusion Model
contentvec768l12	512*30	100	VCTK

m4singer | HuggingFace | | contentvec768l12 | 512*20 | 200 | VCTK

m4singer | HuggingFace |

• The experiment found that the Naive model has f0 range issues on small data. Please prioritize fine-tuning the Naive model with fewer steps or directly using the infinite range ddsp model.

2.3 Naive pre training model and DDSP pre training model matched with 2.2

Units Encoder	Network size	Datasets	Type	Naive Model
contentvec768l12	3*256	VCTK

m4singer | Naive | HuggingFace |

• The pre training naive model can also be used for the previous naive model of the complete Diffusion model. And when fine-tuning the Shallow model, it is recommended to include thedecay_step in the configuration (such as 10000).

3. Training with Pretrained Models:

1. We welcome pull requests for multi-speaker pretrained models (please use datasets that are authorized for open-source training).
2. The pretrained models are mentioned above. Please note that the model must use the same encoder.
3. Place the pretrained model named model_0.pt in the model export folder specified by the "expdir: exp/*" parameter in config.yaml. If the folder doesn't exist, create a new one. The program will automatically load the pretrained model from this folder.
4. Start training just like training without pretrained model.

4.1. Naive model and Combo model

Naive model

The naive model is a lightweight svc model, which can be used as a precursor to shallow diffusion. The training method is consistent with the Diffusion model. The example configuration file is located in configs/config_naive.yaml。The pretreatment required is the same as the Diffusion model.

python train.py -c configs/config_naive.yaml

When inferring, use-nmodelto point to the model file for use, and i