The rectified linear activation function, or ReLU activation function, is perhaps the most common function used for hidden layers.
It is common because it is both simple to implement and effective at overcoming the limitations of other previously popular activation functions, such as Sigmoid and Tanh. Specifically, it is less susceptible to vanishing gradients that prevent deep models from being trained, although it can suffer from other problems like saturated or “dead” units.
The ReLU function is calculated as follows:
- max(0.0, x)
This means that if the input value (x) is negative, then a value 0.0 is returned, otherwise, the value is returned.
We can get an intuition for the shape of this function with the worked example below.
example plot for the relu activation function
from matplotlib import pyplot
rectified linear function
def rectified(x):
return max(0.0, x)
define input data
inputs = [x for x in range(-10, 10)]
calculate outputs
outputs = [rectified(x) for x in inputs]
plot inputs vs outputs
pyplot.plot(inputs, outputs)
pyplot.show()
Running the example calculates the outputs for a range of values and creates a plot of inputs versus outputs.