Computes and stores the average and current value

normalization layer

contrastive loss, corresponding to Eq (18)

Convolutional regression for FitNet

Distilling the Knowledge in a Neural Network

Embedding module

Linear Embedding

Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons code: https://github.com/bhheo/AB_distillation

From: https://hips.seas.harvard.edu/blog/2013/03/03/the-alias-method-efficient-sampling-with-many-discrete-outcomes/

Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer code: https://github.com

CIFAR100Instance Dataset.

CIFAR100Instance+Sample Dataset

CRD Loss function includes two symmetric parts: (a) using teacher as anchor, choose positive and negatives over the student side (b) using

Connect for Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons

A Comprehensive Overhaul of Feature Distillation (ICCV 2019)

memory buffer that supplies large amount of negative samples.

Correlation Congruence for Knowledge Distillation, ICCV 2019. The authors nicely shared the code with me. I restructured their code to be com

A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning

Paraphrasing Complex Network: Network Compression via Factor Transfer, NeurIPS 2018

flatten module

Fitnets: hints for thin deep nets, ICLR 2015

: Folder datasets which returns the index of the image as well::

: Folder datasets which returns (img, label, index, contrast_index):

Self-supervised Knowledge Distillation using Singular Value Decomposition original Tensorflow code: https://github.com/sseung0703/SSKD_SVD

mobilenetV2

like what you like: knowledge distill via neuron selectivity transfer

normalization layer

Probabilistic Knowledge Transfer for deep representation learning Code from author: https://github.com/passalis/probabilistic_kt

Paraphrasing Complex Network: Network Compression via Factor Transfer

Relational Knowledge Disitllation, CVPR2019

Similarity-Preserving Knowledge Distillation, ICCV2019, verified by original author

Variational Information Distillation for Knowledge Transfer (CVPR 2019), code from author: https://github.com/ssahn0215/variational-information-di

Embedding module

non-linear embed by MLP

pool and embed

Simple Linear Regression for hints