Directional diffusion models

NeurIPS 2023

Run Yang¹, Yuling Yang¹, Fan Zhou¹, Qiang Sun²

¹Shanghai University of Finance and Economics, ²University of Toronto

We introduce a novel class of models termed directional diffusion models (DDM), which adopt data-dependent, anisotropic, and directional noises in the forward diffusion process. This code is an implementation of DDM on 12 public graph datasets.

Graph classification datasets

• IMDB-B
• IMDB-M
• COLLAB
• REDDIT-B
• PROTEINS
• MUTAG

Node classification datasets

• CORA
• Citeseer
• PubMed
• Ogbn-arxiv
• Amazon-Computer
• Amazon-Photo

Framework

Usage

conda create -n ddm python=3.8
conda activate ddm
cd ddm-nni
pip install -r requirements.txt

cd to EXP path(MUTAG for example)

cd GraphExp
python main_graph.py --yaml_dir ./yamls/MUTAG.yaml

In view of the sensitivity of diffusion method to hyperparameters, it is recommended to use hyperparameter search methods like NNI to achieve better results

Performance

Directional noise v.s. white noise

Graph classification(F1-score)

	IMDB-B	IMDB-M	COLLAB	REDDIT-B	PROTEINS	MUTAG
GIN[1]	75.1±5.1	52.3±2.8	80.2±1.9	92.4±2.5	76.2±2.8	89.4±5.6
DiffPool[2]	72.6±3.9	-	78.9±2.3	92.1±2.6	75.1±2.3	85.0±10.3
Infograph[3]	73.03±0.87	49.69±0.53	70.65±1.13	82.50±1.42	74.44±0.31	89.01±1.13
GraphCL[4]	71.14±0.44	48.58±0.67	71.36±1.15	89.53±0.84	74.39±0.45	86.80±1.34
JOAO[5]	70.21±3.08	49.20±0.77	69.50±0.36	85.29±1.35	74.55±0.41	87.35±1.02
GCC[6]	72	49.4	78.9	89.8	-	-
MVGRL[7]	74.20±0.70	51.20±0.50	-	84.50±0.60	-	89.70±1.10
GraphMAE[8]	75.52±0.66	51.63±0.52	80.32±0.46	88.01±0.19	75.30±0.39	88.19±1.26
DDM	76.40±0.22	52.53±0.31	81.72±0.31	89.15±1.3	75.74±0.50	91.51±1.45
### Node classification(F1-score)
Dataset	Cora	Citeseer	PubMed	Ogbn-arxiv	Computer	Photo
:---:	:----:	:----:	:----:	:------:	:------:	:---:
GAT	83.0 ± 0.7	72.5 ± 0.7	79.0 ± 0.3	72.10 ± 0.13	86.93 ± 0.29	92.56 ± 0.35
DGI[9]	82.3 ± 0.6	71.8 ± 0.7	76.8 ± 0.6	70.34 ± 0.16	83.95 ± 0.47	91.61 ± 0.22
MVGRL[7]	83.5 ± 0.4	73.3 ± 0.5	80.1 ± 0.7	-	87.52 ± 0.11	91.74 ± 0.07
BGRL[10]	82.7 ± 0.6	71.1 ± 0.8	79.6 ± 0.5	71.64 ± 0.12	89.68 ± 0.31	92.87 ± 0.27
InfoGCL[11]	83.5 ± 0.3	73.5 ± 0.4	79.1 ± 0.2	-	-	-
CCA-SSG[12]	84.0 ± 0.4	73.1 ± 0.3	81.0 ± 0.4	71.24 ± 0.20	88.74 ± 0.28	93.14 ± 0.14
GPT-GNN[13]	80.1 ± 1.0	68.4 ± 1.6	76.3 ± 0.8	-	-	-
GraphMAE[8]	84.2 ± 0.4	73.4 ± 0.4	81.1 ± 0.4	71.75 ± 0.17	88.63 ± 0.17	93.63 ± 0.22
DDM	83.4±0.2	74.3±0.3	81.7±0.8	71.29±0.18	90.56±0.21	95.09±0.18

Citations

[1]:Xu, K., Hu, W., Leskovec, J., and Jegelka, S. (2018). How powerful are graph neural networks? arXiv preprint arXiv:1810.00826.

[2]:Ying, Z., You, J., Morris, C., Ren, X., Hamilton, W., and Leskovec, J. (2018). Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31.

[3]:Sun, F.-Y., Hoffmann, J., Verma, V., and Tang, J. (2019). Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908.01000.

[4]:You, Y., Chen, T., Sui, Y., Chen, T., Wang, Z., and Shen, Y. (2020). Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812–5823.

[5]:You, Y., Chen, T., Shen, Y., and Wang, Z. (2021). Graph contrastive learning automated. In International Conference on Machine Learning, pages 12121–12132. PMLR.

[6]:Qiu, J., Chen, Q., Dong, Y., Zhang, J., Yang, H., Ding, M., Wang, K., and Tang, J. (2020). Gcc: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1150–1160.

[7]:Hassani, K. and Khasahmadi, A. H. (2020). Contrastive multi-view representation learning on graphs. In International conference on machine learning, pages 4116–4126. PMLR.

[8]:Hou, Z., Liu, X., Dong, Y., Wang, C., Tang, J., et al. (2022). Graphmae: Self-supervised masked graph autoencoders. arXiv preprint arXiv:2205.10803.

[9]:Velickovic, P., Fedus, W., Hamilton, W. L., Liò, P., Bengio, Y., and Hjelm, R. D. (2019). Deep graph infomax. ICLR (Poster), 2(3):4.

[10]:Thakoor, S., Tallec, C., Azar, M. G., Azabou, M., Dyer, E. L., Munos, R., Veliˇckovi ́c, P., and Valko, M. (2021). Large-scale representation learning on graphs via bootstrapping. arXiv preprint arXiv:2102.06514.

[11]:Xu, D., Cheng, W., Luo, D., Chen, H., and Zhang, X. (2021). Infogcl: Information-aware graph contrastive learning. Advances in Neural Information Processing Systems, 34:30414–30425.

[12]:Zhang, H., Wu, Q., Yan, J., Wipf, D., and Yu, P. S. (2021). From canonical correlation analysis to self-supervised graph neural networks. Advances in Neural Information Processing Systems, 34:76–89.

[13]:Hu, Z., Dong, Y., Wang, K., Chang, K.-W., and Sun, Y. (2020b). Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1857–1867.