The Curve After ReLU: Why GELU, SiLU and SwiGLU Promise Better Generalization

StatQuest host Josh Starmer sets out to explain three activation functions, GELU, SiLU and SwiGLU, to anyone who already knows ReLU, and the headline is simple: the first two differ from ReLU only subtly, the third can morph into far wilder shapes, yet all three share a family trait of resisting the urge to memorize training data.

The visible differences are small but precise: where ReLU flatlines at zero for every negative input and kinks sharply at the origin, GELU and SiLU dip slightly below zero between roughly minus two and zero, round off the corner into a smooth differentiable bend, and return values a touch smaller than the input on the positive side.

SwiGLU plays a different game: it can mimic a smoothed ReLU, but its extra parameters let it bend into shapes its cousins cannot reach, which is exactly why the episode treats it as the flexible heavyweight of the trio rather than a mere variant.

The payoff the episode promises is generalization: a large ReLU network draped over scattered training points hugs every dot including the noise, while the same capacity built on the new activations traces a calmer middle path that is likelier to predict unseen points well.

The story rewinds to before 2010, when sigmoid ruled: its zero-to-one output echoed a neuron either firing or silent, its S-curve mimicked the transition between those states, and its smoothness everywhere made backpropagation straightforward, so shallow networks trained happily.

Depth broke the spell: far from zero the sigmoid slope collapses toward nothing, gradients shrink to whispers, each optimization step crawls, and a network that should have traced the ideal wavy fit stalls into a timid curve that misses the data, the classic underfit.

Around 2010 ReLU took over with a bent line almost suspiciously simple: negatives become zero, positives pass through unchanged, the slope is either zero or one, so large inputs keep gradients alive and deep stacks finally become trainable, a run crowned in 2017 inside the first Transformer.

But capacity became the new trap: a giant ReLU network does not underfit anymore, it overfits, wrapping every quirk of the sample, and shrinking the network is no real answer since nobody can test a hundred thousand sizes one unit at a time to find the sweet spot.

Dropout was the first systematic fix the episode recalls: during training random subsets of units are silenced, each step effectively trains a thinner network, and at prediction time the full model behaves like an average over all those thin networks, pulling the wobbly ReLU fit closer to the ideal; but the fix has two costs, machinery bolted onto training and blindness to signal strength, since dropout kills units without asking whether their input is large or small, which motivates making survival odds depend on the input itself.

The derivation starts from a toy one-input network under fifty-percent dropout, where an input of minus two averages to minus one, then replaces the coin flip with a curve: the Gaussian cumulative curve maps minus two to a two-percent survival chance for an average of minus zero point zero four, minus one to sixteen percent for minus zero point one six, with zero, zero point eight four at one, and one point nine six at two falling on the emerging GELU shape, compactly written as the input times its own survival probability.

Swapping the bell-curve mapping for the familiar sigmoid yields SiLU, a near-twin of GELU that thrives in similar settings and leaves practitioners choosing mostly by ecosystem, while SwiGLU adds a learned weight, a Swish-like branch and a second projection multiplied together through a gating junction, a design whose own paper offers no causal story, only results, plus the host's two hunches that costlier units force leaner networks and that trainable shapes let the network decide for itself, a bet Meta's model family visibly took.