🎵 Speech Separation Paper Tutorial

🚀 A comprehensive collection of speech separation papers, models, and resources from 2016-2025

📋 Table of Contents

• 📊 Overview & Statistics
• 📈 Model Timeline
• 🏆 Performance Comparison
• 🔬 Model Categories
• 🎯 Deterministic vs Generative
• 🏗️ Network Architecture
• 🎭 Mask vs Mapping
• 🧠 Learning Methods
• 👥 Speaker Knowledge
• 📅 Papers by Year
• 🎬 Multi-Modal Speech Separation
• 📏 Evaluation Metrics
• 📚 Datasets
• 🎥 Video Tutorial

📊 Overview & Statistics

📈 Model Timeline

Figure 1: Speech separation models development timeline (2016-2025)

📊 Parameter vs Performance Analysis

Figure 2: Model parameters vs WSJ0-2Mix performance scatter plot

🔢 Statistics Summary

• Total Models: 69
• Years Covered: 2016-2025 (10 years)
• Deterministic Models: 60 (87%)
• Generative Models: 9 (13%)
• Known Speaker Models: 58 (84%)
• Unknown Speaker Models: 11 (16%)

🏆 Performance Comparison

🥇 Top Performing Models

WSJ0-2Mix Dataset (SI-SNRi)

Rank	Model	Year	SI-SNRi (dB)	SDRi (dB)	Params (M)	Code
1	SepTDA	2024	24.0	23.9	21.2	-
2	SFSRNet	2022	24.0	24.1	59.0	🔗
3	MossFormer2	2024	24.1	-	55.7	🔗
4	Separate And Diffuse	2023	23.9	-	-	🔗
5	QDPN	2022	23.6	-	200.0	-

WHAM! Dataset (SI-SNRi)

Rank	Model	Year	SI-SNRi (dB)	SDRi (dB)	Params (M)	Code
1	MossFormer2	2024	18.1	-	55.7	🔗
2	SPMamba	2024	17.4	17.6	6.1	🔗
3	MossFormer	2023	17.3	-	42.1	🔗
4	SepFormer	2021	16.4	-	26.0	🔗
5	WaveSplit	2021	16.0	16.5	29.0	🔗

LibriMix Dataset (SI-SNRi)

Rank	Model	Year	SI-SNRi (dB)	SDRi (dB)	Params (M)	Code
1	Separate And Diffuse	2023	21.5	-	-	🔗
2	MossFormer2	2024	21.7	-	55.7	🔗
3	SPMamba	2024	19.9	20.4	6.1	🔗
4	MossFormer	2023	19.7	-	42.1	🔗
5	TFPSNet	2022	19.7	19.9	2.7	-

🔬 Model Categories

🎯 Deterministic vs Generative

🔧 Deterministic Models (60 models)

Deterministic models use fixed neural network architectures to directly predict separation masks or mappings.

Representative Models: - SepTDA (2024): 24.0 dB SI-SNRi on WSJ0-2Mix - MossFormer2 (2024): 24.1 dB SI-SNRi on WSJ0-2Mix - SPMamba (2024): State-space model approach - SepFormer (2021): Pure attention-based architecture

🎨 Generative Models (9 models)

Generative models use probabilistic approaches like GANs, VAEs, or diffusion models.

Representative Models: - EDSep (2025): Diffusion-based method, 15.9 dB SI-SNRi - Fast-GeCo (2024): Fast generative correction - SepDiff (2023): Denoising diffusion model - DiffSep (2023): Diffusion-based generative separation

🏗️ Network Architecture

🔄 Dual-path Architecture (22 models)

Dual-path networks process sequences in both intra-chunk and inter-chunk dimensions.

Key Models: - SepTDA (2024): Transformer decoder-based attractor - SPMamba (2024): State-space model integration - TF-GridNet (2023): Full and sub-band modeling - SepFormer (2021): Pure transformer architecture - DPRNN (2020): Foundational dual-path RNN

🌊 Conv-TasNet Architecture (20 models)

Time-domain audio separation networks using 1D convolutions.

Key Models: - MossFormer2 (2024): 55.7M parameters - MossFormer (2023): Gated single-head transformer - ConvTasNet (2019): Original Conv-TasNet architecture - TaSNet (2018): Time-domain audio separation

🏗️ U-Net Architecture (10 models)

Encoder-decoder architectures with skip connections.

Key Models: - EDSep (2025): Diffusion-based U-Net - S4M (2023): Neural state-space model - TDANet (2022): Top-down attention - A-FRCNN (2021): Asynchronous fully recurrent CNN

🎭 Mask vs Mapping

🎭 Mask-based Methods (39 models)

Predict multiplicative masks to separate sources.

Advantages: - Interpretable separation process - Preserves phase information - Stable training

🗺️ Mapping-based Methods (24 models)

Directly map mixed signals to separated sources.

Advantages: - End-to-end optimization - Potentially better reconstruction - More flexible architectures

🧠 Learning Methods

🎯 Predictive Methods (58 models)

Supervised learning with known target separations.

🔄 Clustering Methods (6 models)

Use embedding clustering for speaker separation: - Chimera++ Network (2018) - DANet (2017) - DPCL (2016)

🎲 Unsupervised Methods (4 models)

Learn separation without paired training data: - UNSSOR (2023) - TS-MixIT (2021) - MixIT (2020) - VAE (2021)

👥 Speaker Knowledge

✅ Known Speaker Models (58 models)

Assume fixed number of speakers (typically 2).

❓ Unknown Speaker Models (11 models)

Handle variable number of speakers: - SepTDA (2024): Transformer decoder-based attractor - SepEDA (2022): Encoder-decoder attractors - VSUNOS (2020): Voice separation for unknown speakers - Multi-Decoder DPRNN (2021)

📅 Papers by Year

🚀 2025 (1 model)

Model	Paper	SI-SNRi (WSJ0)	Params (M)	Code	Paper
EDSep	EDSep: An Effective Diffusion-Based Method for Speech Source Separation	15.9	-	-	📄

🔥 2024 (10 models)

Model	Paper	SI-SNRi (WSJ0)	Params (M)	Code	Paper
ReSepFormer	Resource-Efficient Separation Transformer	18.6	8.0	🔗	📄
Conv-TasNet GAN	Exploring GANs With Conv-TasNet	-	-	🔗	-
SepTDA	Boosting Unknown-Number Speaker Separation	24.0	21.2	-	📄
SPMamba	SPMamba: State-space model is all you need	22.5	6.1	🔗	📄
Fast-GeCo	Noise-robust Speech Separation with Fast Generative Correction	-	-	🔗	📄
DIP	Speech Separation With Pretrained Frontend	-	-	-	📄
TIGER	TIGER: Time-frequency Interleaved Gain Extraction	-	0.8	🔗	📄
CodecSS	Speech Separation using Neural Audio Codecs	-	-	-	📄
TCodecSS	Towards Audio Codec-based Speech Separation	-	-	-	📄
MossFormer2	MossFormer2: Combining Transformer and RNN-Free Recurrent Network	24.1	55.7	🔗	📄

⭐ 2023 (10 models)

Model	Paper	SI-SNRi (WSJ0)	Params (M)	Code	Paper
SepDiff	Sepdiff: Speech separation based on denoising diffusion	-	-	-	📄
S4M	A Neural State-Space Model Approach	20.5	3.6	🔗	📄
HuBERT	Cocktail Hubert: Generalized Self-Supervised Pre-Training	-	-	-	📄
Diff-Refiner	Diffusion-based signal refiner for speech separation	-	-	-	📄
CycleGAN-SS	Cycle GAN-Based Audio Source Separation	-	-	-	📄
pSkiM	Predictive Skim: Contrastive Predictive Coding	15.5	8.5	-	📄
PGSS	PGSS: Pitch-Guided Speech Separation	-	-	-	📄
Separate And Diffuse	Using a Pretrained Diffusion Model	23.9	-	🔗	📄
DiffSep	Diffusion-Based Generative Speech Source Separation	14.3	-	🔗	📄
TF-GridNet	Integrating Full- and Sub-Band Modeling	23.5	14.5	🔗	📄
UNSSOR	Unsupervised Neural Speech Separation	-	-	-	📄
MossFormer	Pushing the Performance Limit of Monaural Speech Separation	22.8	42.1	🔗	📄

🎯 2022 (7 models)

Model	Paper	SI-SNRi (WSJ0)	Params (M)	Code	Paper
SepEDA	Speech Separation for Unknown Number of Speakers	21.2	12.5	-	📄
SSL-SS	Investigating Self-Supervised Learning	-	-	-	📄
SkiM	Skipping Memory Lstm for Low-Latency	18.3	5.9	🔗	📄
TDANet	Efficient encoder-decoder architecture	18.6	2.3	🔗	📄
MTDS	Efficient Monaural Speech Separation	21.5	4.0	-	📄
QDPN	Quasi-dual-path Network	23.6	200.0	-	📄
SFSRNet	Super-resolution for Single-Channel Audio	24.0	59.0	🔗	📄
TFPSNet	Time-Frequency Domain Path Scanning Network	21.1	2.7	-	📄

🌟 2021 (13 models)

Model	Paper	SI-SNRi (WSJ0)	Params (M)	Code	Paper
Unknow-SS	Single channel voice separation for unknown number	19.4	-	-	📄
VAE	Unsupervised Blind Source Separation with VAE	-	-	🔗	📄
A-FRCNN	Speech Separation Using Asynchronous Fully Recurrent CNN	18.3	6.1	🔗	📄
Sandglasset	A Light Multi-Granularity Self-Attentive Network	20.8	2.3	🔗	📄
CDGAN	Generative adversarial networks for single channel separation	-	-	-	📄
SepFormer	Attention Is All You Need In Speech Sepa