Activation Functions
Visualize and compare common neural network activation functions.
Free theory articleVisualize and compare common neural network activation functions.
Concept & interpretation
Activation functions are mathematical equations attached to each neuron in a neural network, determining whether the neuron should be activated. Crucially, they introduce non-linearities into the network. Without non-linear activation functions, regardless of how many layers a neural network has, it would simply behave as a single-layer perceptron capable only of solving linearly separable problems.