A hidden layer in a neural network is a layer that receives input from another layer (such as another hidden layer or an input layer) and provides output to another layer (such as another hidden layer or an output layer).
A hidden layer does not directly contact input data or produce outputs for a model, at least in general.
A neural network may have zero or more hidden layers.
Typically, a differentiable nonlinear activation function is used in the hidden layers of a neural network. This allows the model to learn more complex functions than a network trained using a linear activation function.
In order to get access to a much richer hypothesis space that would benefit from deep representations, you need a non-linearity, or activation function.
— Page 72, Deep Learning with Python, 2017.
There are perhaps three activation functions you may want to consider for use in hidden layers; they are:
- Rectified Linear Activation (ReLU)
- Logistic (Sigmoid)
- Hyperbolic Tangent (Tanh)
This is not an exhaustive list of activation functions used for hidden layers, but they are the most commonly used.