The output layer is the layer in a neural network model that directly outputs a prediction.
All feed-forward neural network models have an output layer.
There are perhaps three activation functions you may want to consider for use in the output layer; they are:
Linear
Logistic (Sigmoid)
Softmax
This is not an exhaustive list of activation functions used for output layers, but they are the most commonly used.
Let’s take a closer look at each in turn.
Linear Output Activation Function
The linear activation function is also called “identity” (multiplied by 1.0) or “no activation.”
This is because the linear activation function does not change the weighted sum of the input in any way and instead returns the value directly.
We can get an intuition for the shape of this function with the worked example below.
example plot for the linear activation function
from matplotlib import pyplot
linear activation function
def linear(x):
return x
define input data
inputs = [x for x in range(-10, 10)]
calculate outputs
outputs = [linear(x) for x in inputs]
plot inputs vs outputs
pyplot.plot(inputs, outputs)
pyplot.show()
Running the example calculates the outputs for a range of values and creates a plot of inputs versus outputs.
We can see a diagonal line shape where inputs are plotted against identical outputs.
Plot of Inputs vs. Outputs for the Linear Activation Function
Plot of Inputs vs. Outputs for the Linear Activation Function
Target values used to train a model with a linear activation function in the output layer are typically scaled prior to modeling using normalization or standardization transforms.