IEEE Transactions on Artificial Intelligence 2022
SparseShortcutTopology
On a Sparse Shortcut Topology of Artificial Neural Networks
Current or former lab member. Select a name to see all of their projects.
Abstract
Shortcut connections feed the outputs of earlier layers into later layers, and they work remarkably well. This paper studies a sparse shortcut topology for its expressivity and generalizability. It shows that the topology lets a one-neuron-wide deep network approximate any univariate continuous function, and presents a width-bounded universal approximator.
Generalization bound theory, comparisons with ResNet and DenseNet on benchmarks, and saliency map analysis indicate that the topology generalizes well and suggest ways to design new architectures.
What’s in the repository
- Expressibility experiments in the neural tangent kernel setting
- Generalizability experiments on CIFAR-100, Tiny ImageNet and ImageNet
- Interpretability experiments with saliency maps
Citation
@article{fan2022sparse,
title = {On a Sparse Shortcut Topology of Artificial Neural Networks},
author = {Fan, Fenglei and Wang, Dayang and Guo, Hengtao and Zhu, Qikui and Yan, Pingkun and Wang, Ge and Yu, Hengyong},
journal = {IEEE Transactions on Artificial Intelligence},
volume = {3},
number = {4},
pages = {595--608},
year = {2022},
doi = {10.1109/TAI.2021.3128132}
}