IEEE Transactions on Neural Networks and Learning Systems 2025

ReLinear

On Expressivity and Trainability of Quadratic Networks

Fenglei Fan, Mengzhou Li, Fei Wang, Rongjie Lai, Ge Wang

Current or former lab member. Select a name to see all of their projects.

Method overview of ReLinear
The ReLinear training strategy, from the repository: quadratic terms start from a conventional neuron and are learned gradually.

Abstract

Quadratic neurons replace the inner product of a conventional neuron with a quadratic function. Networks built from them show promising results, but their advantage in expressivity had not been fully explained, and they are more prone to training collapse. Using spline theory and a measure from algebraic geometry, this paper proves two theorems showing that quadratic networks are more expressive than their conventional counterparts.

To stabilize training, the paper proposes ReLinear. Each quadratic neuron is initialized so that it behaves like a conventional neuron, and the quadratic terms are then regularized during training so that they are learned gradually and adaptively. Experiments on popular datasets support the analysis.

What’s in the repository

  • Training quadratic networks with ReLinear
  • A compact quadratic network variant
  • A variant combining ReLinear with ReZero

Citation

@article{fan2025expressivity,
  title   = {On Expressivity and Trainability of Quadratic Networks},
  author  = {Fan, Fenglei and Li, Mengzhou and Wang, Fei and Lai, Rongjie and Wang, Ge},
  journal = {IEEE Transactions on Neural Networks and Learning Systems},
  volume  = {36},
  number  = {1},
  pages   = {1228--1242},
  year    = {2025},
  doi     = {10.1109/TNNLS.2023.3331380}
}
Back to projects