Softmax Output Activation Function
The softmax function outputs a vector of values that sum to 1.0 that can be interpreted as probabilities of class membership.
It is related to the argmax function that outputs a 0 for all options and 1 for the chosen option. Softmax is a “softer” version of argmax that allows a probability-like output of a winner-take-all function.
As such, the input to the function is a vector of real values and the output is a vector of the same length with values that sum to 1.0 like probabilities.
The softmax function is calculated as follows:
e^x / sum(e^x)
Where x is a vector of outputs and e is a mathematical constant that is the base of the natural logarithm.
You can learn more about the details of the Softmax function in this tutorial:
Softmax Activation Function with Python
We cannot plot the softmax function, but we can give an example of calculating it in Python.
from numpy import exp
softmax activation function
def softmax(x):
return exp(x) / exp(x).sum()
define input data
inputs = [1.0, 3.0, 2.0]
calculate outputs
outputs = softmax(inputs)
report the probabilities
print(outputs)
report the sum of the probabilities
print(outputs.sum())
Running the example calculates the softmax output for the input vector.
We then confirm that the sum of the outputs of the softmax indeed sums to the value 1.0.
[0.09003057 0.66524096 0.24472847]
1.0
Target labels used to train a model with the softmax activation function in the output layer will be vectors with 1 for the target class and 0 for all other classes.