Activation functions are a critical part of the design of a neural network.
The choice of activation function in the hidden layer will control how well the network model learns the training dataset. The choice of activation function in the output layer will define the type of predictions the model can make.
As such, a careful choice of activation function must be made for each deep learning neural network project.
Sometimes the activation function is called a “transfer function .” If the output range of the activation function is limited, then it may be called a “squashing function .” Many activation functions are nonlinear and may be referred to as the “nonlinearity ” in the layer or the network design.
The choice of activation function has a large impact on the capability and performance of the neural network, and different activation functions may be used in different parts of the model.
Technically, the activation function is used within or after the internal processing of each node in the network, although networks are designed to use the same activation function for all nodes in a layer.