The normalized xavier initialization method is calculated as a random number with a uniform probability distribution (U) between the range -(sqrt(6)/sqrt(n + m)) and sqrt(6)/sqrt(n + m), where n us the number of inputs to the node (e.g. number of nodes in the previous layer) and m is the number of outputs from the layer (e.g. number of nodes in the current layer).
weight = U [-(sqrt(6)/sqrt(n + m)), sqrt(6)/sqrt(n + m)]
We can implement this directly in Python as we did in the previous section and summarize the statistical summary of 1,000 generated weights.
The complete example is listed below.
example of the normalized xavier weight initialization
from math import sqrt
from numpy import mean
from numpy.random import rand
number of nodes in the previous layer
n = 10
number of nodes in the next layer
m = 20
calculate the range for the weights
lower, upper = -(sqrt(6.0) / sqrt(n + m)), (sqrt(6.0) / sqrt(n + m))
generate random numbers
numbers = rand(1000)
scale to the desired range
scaled = lower + numbers * (upper - lower)
summarize
print(lower, upper)
print(scaled.min(), scaled.max())
print(scaled.mean(), scaled.std())
example of the normalized xavier weight initialization
from math import sqrt
from numpy import mean
from numpy.random import rand
number of nodes in the previous layer
n = 10
number of nodes in the next layer
m = 20
calculate the range for the weights
lower, upper = -(sqrt(6.0) / sqrt(n + m)), (sqrt(6.0) / sqrt(n + m))
generate random numbers
numbers = rand(1000)
scale to the desired range
scaled = lower + numbers * (upper - lower)
summarize
print(lower, upper)
print(scaled.min(), scaled.max())
print(scaled.mean(), scaled.std())
Running the example generates the weights and prints the summary statistics.
We can see that the bounds of the weight values are about -0.447 and 0.447. These bounds would become wider with fewer inputs and more narrow with more inputs.
We can see that the generated weights respect these bounds and that the mean weight value is close to zero with the standard deviation close to 0.17.
-0.44721359549995787 0.44721359549995787
-0.4447861894315135 0.4463641245392874
-0.01135636099916006 0.2581340352889168
-0.44721359549995787 0.44721359549995787
-0.4447861894315135 0.4463641245392874
-0.01135636099916006 0.2581340352889168
It can also help to see how the spread of the weights changes with the number of inputs.
For this, we can calculate the bounds on the weight initialization with different numbers of inputs from 1 to 100 and a fixed number of 10 outputs and plot the result.
The complete example is listed below.
plot of the bounds of normalized xavier weight initialization for different numbers of inputs
from math import sqrt
from matplotlib import pyplot
define the number of inputs from 1 to 100
values = [i for i in range(1, 101)]
define the number of outputs
m = 10
calculate the range for each number of inputs
results = [1.0 / sqrt(n + m) for n in values]
create an error bar plot centered on 0 for each number of inputs
pyplot.errorbar(values, [0.0 for _ in values], yerr=results)
pyplot.show()
plot of the bounds of normalized xavier weight initialization for different numbers of inputs
from math import sqrt
from matplotlib import pyplot
define the number of inputs from 1 to 100
values = [i for i in range(1, 101)]
define the number of outputs
m = 10
calculate the range for each number of inputs
results = [1.0 / sqrt(n + m) for n in values]
create an error bar plot centered on 0 for each number of inputs
pyplot.errorbar(values, [0.0 for _ in values], yerr=results)
pyplot.show()
Running the example creates a plot that allows us to compare the range of weights with different numbers of input values.