Gradient Descent with Momentum

One-Dimensional Test Problem

First, let’s define an optimization function.

We will use a simple one-dimensional function that squares the input and defines the range of valid inputs from -1.0 to 1.0.

The objective() function below implements this function.

objective function

def objective(x):
return x**2.0

objective function

def objective(x):
return x**2.0
We can then sample all inputs in the range and calculate the objective function value for each.

…

define range for input

r_min, r_max = -1.0, 1.0

sample input range uniformly at 0.1 increments

inputs = arange(r_min, r_max+0.1, 0.1)

compute targets

results = objective(inputs)

…

define range for input

r_min, r_max = -1.0, 1.0

sample input range uniformly at 0.1 increments

inputs = arange(r_min, r_max+0.1, 0.1)

compute targets

results = objective(inputs)
Finally, we can create a line plot of the inputs (x-axis) versus the objective function values (y-axis) to get an intuition for the shape of the objective function that we will be searching.

…

create a line plot of input vs result

pyplot.plot(inputs, results)

show the plot

pyplot.show()

…

create a line plot of input vs result

pyplot.plot(inputs, results)

show the plot

pyplot.show()
The example below ties this together and provides an example of plotting the one-dimensional test function.

plot of simple function

from numpy import arange
from matplotlib import pyplot

objective function

def objective(x):
return x**2.0

define range for input

r_min, r_max = -1.0, 1.0

sample input range uniformly at 0.1 increments

inputs = arange(r_min, r_max+0.1, 0.1)

compute targets

results = objective(inputs)

create a line plot of input vs result

pyplot.plot(inputs, results)

show the plot

pyplot.show()

plot of simple function

from numpy import arange
from matplotlib import pyplot

objective function

def objective(x):
return x**2.0

define range for input

r_min, r_max = -1.0, 1.0

sample input range uniformly at 0.1 increments

inputs = arange(r_min, r_max+0.1, 0.1)

compute targets

results = objective(inputs)

create a line plot of input vs result

pyplot.plot(inputs, results)

show the plot

pyplot.show()
Running the example creates a line plot of the inputs to the function (x-axis) and the calculated output of the function (y-axis).

We can see the familiar U-shape called a parabola.

Line Plot of Simple One Dimensional Function