The sigmoid activation function is also called the logistic function.
It is the same function used in the logistic regression classification algorithm.
The function takes any real value as input and outputs values in the range 0 to 1. The larger the input (more positive), the closer the output value will be to 1.0, whereas the smaller the input (more negative), the closer the output will be to 0.0.
The sigmoid activation function is calculated as follows:
1.0 / (1.0 + e^-x)
Where e is a mathematical constant, which is the base of the natural logarithm.
We can get an intuition for the shape of this function with the worked example below.
example plot for the sigmoid activation function
from math import exp
from matplotlib import pyplot
sigmoid activation function
def sigmoid(x):
return 1.0 / (1.0 + exp(-x))
define input data
inputs = [x for x in range(-10, 10)]
calculate outputs
outputs = [sigmoid(x) for x in inputs]
plot inputs vs outputs
pyplot.plot(inputs, outputs)
pyplot.show()
Running the example calculates the outputs for a range of values and creates a plot of inputs versus outputs.
We can see the familiar S-shape of the sigmoid activation function.
Plot of Inputs vs. Outputs for the Sigmoid Activation Function.
Plot of Inputs vs. Outputs for the Sigmoid Activation Function.
When using the Sigmoid function for hidden layers, it is a good practice to use a “Xavier Normal” or “Xavier Uniform” weight initialization (also referred to Glorot initialization, named for Xavier Glorot) and scale input data to the range 0-1 (e.g. the range of the activation function) prior to training.