The “xavier” weight initialization was found to have problems when used to initialize networks that use the rectified linear (ReLU) activation function.
As such, a modified version of the approach was developed specifically for nodes and layers that use ReLU activation, popular in the hidden layers of most multilayer Perceptron and convolutional neural network models.
The current standard approach for initialization of the weights of neural network layers and nodes that use the rectified linear (ReLU) activation function is called “he” initialization.
He Weight Initialization
The he initialization method is calculated as a random number with a Gaussian probability distribution (G) with a mean of 0.0 and a standard deviation of sqrt(2/n), where n is the number of inputs to the node.
weight = G (0.0, sqrt(2/n))
We can implement this directly in Python.
The example below assumes 10 inputs to a node, then calculates the standard deviation of the Gaussian distribution and calculates 1,000 initial weight values that could be used for the nodes in a layer or a network that uses the ReLU activation function.
After calculating the weights, the calculated standard deviation is printed as are the min, max, mean, and standard deviation of the generated weights.
The complete example is listed below.
example of the he weight initialization
from math import sqrt
from numpy.random import randn
number of nodes in the previous layer
n = 10
calculate the range for the weights
std = sqrt(2.0 / n)
generate random numbers
numbers = randn(1000)
scale to the desired range
scaled = numbers * std
summarize
print(std)
print(scaled.min(), scaled.max())
print(scaled.mean(), scaled.std())
example of the he weight initialization
from math import sqrt
from numpy.random import randn
number of nodes in the previous layer
n = 10
calculate the range for the weights
std = sqrt(2.0 / n)
generate random numbers
numbers = randn(1000)
scale to the desired range
scaled = numbers * std
summarize
print(std)
print(scaled.min(), scaled.max())
print(scaled.mean(), scaled.std())
Running the example generates the weights and prints the summary statistics.
We can see that the bound of the calculated standard deviation of the weights is about 0.447. This standard deviation would become larger with fewer inputs and smaller with more inputs.
We can see that the range of the weights is about -1.573 to 1.433 which is close to the theoretical range of about -1.788 and 1.788, which is four times the standard deviation, capturing 99.7% of observations in the Gaussian distribution. We can also see that the mean and standard deviation of the generated weights are close to the prescribed 0.0 and 0.447 respectively.
0.4472135954999579
-1.5736761136523203 1.433348584081719
-0.00023406487278826836 0.4522609460629265
0.4472135954999579
-1.5736761136523203 1.433348584081719
-0.00023406487278826836 0.4522609460629265
It can also help to see how the spread of the weights changes with the number of inputs.
For this, we can calculate the bounds on the weight initialization with different numbers of inputs from 1 to 100 and plot the result.
The complete example is listed below.
plot of the bounds on he weight initialization for different numbers of inputs
from math import sqrt
from matplotlib import pyplot
define the number of inputs from 1 to 100
values = [i for i in range(1, 101)]
calculate the range for each number of inputs
results = [sqrt(2.0 / n) for n in values]
create an error bar plot centered on 0 for each number of inputs
pyplot.errorbar(values, [0.0 for _ in values], yerr=results)
pyplot.show()
plot of the bounds on he weight initialization for different numbers of inputs
from math import sqrt
from matplotlib import pyplot
define the number of inputs from 1 to 100
values = [i for i in range(1, 101)]
calculate the range for each number of inputs
results = [sqrt(2.0 / n) for n in values]
create an error bar plot centered on 0 for each number of inputs
pyplot.errorbar(values, [0.0 for _ in values], yerr=results)
pyplot.show()
Running the example creates a plot that allows us to compare the range of weights with different numbers of input values.