Tanh Hidden Layer Activation Function

Data Science, AI and ML
#1

The hyperbolic tangent activation function is also referred to simply as the Tanh (also “tanh” and “TanH“) function.

It is very similar to the sigmoid activation function and even has the same S-shape.

The function takes any real value as input and outputs values in the range -1 to 1. The larger the input (more positive), the closer the output value will be to 1.0, whereas the smaller the input (more negative), the closer the output will be to -1.0.

The Tanh activation function is calculated as follows:

(e^x – e^-x) / (e^x + e^-x)
Where e is a mathematical constant that is the base of the natural logarithm.

We can get an intuition for the shape of this function with the worked example below.

example plot for the tanh activation function

from math import exp
from matplotlib import pyplot

tanh activation function

def tanh(x):
return (exp(x) - exp(-x)) / (exp(x) + exp(-x))

define input data

inputs = [x for x in range(-10, 10)]

calculate outputs

outputs = [tanh(x) for x in inputs]

plot inputs vs outputs

pyplot.plot(inputs, outputs)
pyplot.show()
Running the example calculates the outputs for a range of values and creates a plot of inputs versus outputs.

We can see the familiar S-shape of the Tanh activation function.

Plot of Inputs vs. Outputs for the Tanh Activation Function.
Plot of Inputs vs. Outputs for the Tanh Activation Function.

When using the TanH function for hidden layers, it is a good practice to use a “Xavier Normal” or “Xavier Uniform” weight initialization (also referred to Glorot initialization, named for Xavier Glorot) and scale input data to the range -1 to 1 (e.g. the range of the activation function) prior to training.