All Courses
AI Career Planner
1:1 Coaching/Mentoring
Job Board
SQL
HTML/CSS/JS
Coding
Settings
LogoutEvaluating and Tuning MLP Models
The k-fold cross-validation procedure can provide a more reliable estimate of MLP performance, although it can be very slow.
This is because k models must be fit and evaluated. This is not a problem when the dataset size is small, such as the auto insurance dataset.
We can use the KFold class to create the splits and enumerate each fold manually, fit the model, evaluate it, and then report the mean of the evaluation scores at the end of the procedure.
prepare cross validation
kfold = KFold(10)
enumerate splits
scores = list()
for train_ix, test_ix in kfold.split(X, y):
# fit and evaluate the model…
…
…
summarize all scores
print(‘Mean MAE: %.3f (%.3f)’ % (mean(scores), std(scores)))
prepare cross validation
kfold = KFold(10)
enumerate splits
scores = list()
for train_ix, test_ix in kfold.split(X, y):
# fit and evaluate the model…
…
…
summarize all scores
print(‘Mean MAE: %.3f (%.3f)’ % (mean(scores), std(scores)))
We can use this framework to develop a reliable estimate of MLP model performance with a range of different data preparations, model architectures, and learning configurations.
It is important that we first developed an understanding of the learning dynamics of the model on the dataset in the previous section before using k-fold cross-validation to estimate the performance. If we started to tune the model directly, we might get good results, but if not, we might have no idea of why, e.g. that the model was over or under fitting.
If we make large changes to the model again, it is a good idea to go back and confirm that model is converging appropriately.
The complete example of this framework to evaluate the base MLP model from the previous section is listed below.
k-fold cross-validation of base model for the auto insurance regression dataset
from numpy import mean
from numpy import std
from pandas import read_csv
from sklearn.model_selection import KFold
from sklearn.metrics import mean_absolute_error
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
from matplotlib import pyplot
load the dataset
path = ‘https://raw.githubusercontent.com/jbrownlee/Datasets/master/auto-insurance.csv’
df = read_csv(path, header=None)
split into input and output columns
X, y = df.values[:, :-1], df.values[:, -1]
prepare cross validation
kfold = KFold(10)
enumerate splits
scores = list()
for train_ix, test_ix in kfold.split(X, y):
# split data
X_train, X_test, y_train, y_test = X[train_ix], X[test_ix], y[train_ix], y[test_ix]
# determine the number of input features
n_features = X.shape[1]
# define model
model = Sequential()
model.add(Dense(10, activation=‘relu’, kernel_initializer=‘he_normal’, input_shape=(n_features,)))
model.add(Dense(1))
# compile the model
model.compile(optimizer=‘adam’, loss=‘mse’)
# fit the model
model.fit(X_train, y_train, epochs=100, batch_size=8, verbose=0)
# predict test set
yhat = model.predict(X_test)
# evaluate predictions
score = mean_absolute_error(y_test, yhat)
print(’>%.3f’ % score)
scores.append(score)
summarize all scores
print(‘Mean MAE: %.3f (%.3f)’ % (mean(scores), std(scores)))
k-fold cross-validation of base model for the auto insurance regression dataset
from numpy import mean
from numpy import std
from pandas import read_csv
from sklearn.model_selection import KFold
from sklearn.metrics import mean_absolute_error
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
from matplotlib import pyplot
load the dataset
path = ‘https://raw.githubusercontent.com/jbrownlee/Datasets/master/auto-insurance.csv’
df = read_csv(path, header=None)
split into input and output columns
X, y = df.values[:, :-1], df.values[:, -1]
prepare cross validation
kfold = KFold(10)
enumerate splits
scores = list()
for train_ix, test_ix in kfold.split(X, y):
# split data
X_train, X_test, y_train, y_test = X[train_ix], X[test_ix], y[train_ix], y[test_ix]
# determine the number of input features
n_features = X.shape[1]
# define model
model = Sequential()
model.add(Dense(10, activation=‘relu’, kernel_initializer=‘he_normal’, input_shape=(n_features,)))
model.add(Dense(1))
# compile the model
model.compile(optimizer=‘adam’, loss=‘mse’)
# fit the model
model.fit(X_train, y_train, epochs=100, batch_size=8, verbose=0)
# predict test set
yhat = model.predict(X_test)
# evaluate predictions
score = mean_absolute_error(y_test, yhat)
print(’>%.3f’ % score)
scores.append(score)
summarize all scores
print(‘Mean MAE: %.3f (%.3f)’ % (mean(scores), std(scores)))
Running the example reports the model performance each iteration of the evaluation procedure and reports the mean and standard deviation of the MAE at the end of the run.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
In this case, we can see that the MLP model achieved a MAE of about 38.913.
We will use this result as our baseline to see if we can achieve better performance.
27.314
69.577
20.891
14.810
13.412
69.540
25.612
49.508
35.769
62.696
Mean MAE: 38.913 (21.056)
27.314
69.577
20.891
14.810
13.412
69.540
25.612
49.508
35.769
62.696
Mean MAE: 38.913 (21.056)
First, let’s try evaluating a deeper model on the raw dataset to see if it performs any better than a baseline model.
…
define model
model = Sequential()
model.add(Dense(10, activation=‘relu’, kernel_initializer=‘he_normal’, input_shape=(n_features,)))
model.add(Dense(8, activation=‘relu’, kernel_initializer=‘he_normal’))
model.add(Dense(1))
compile the model
model.compile(optimizer=‘adam’, loss=‘mse’)
fit the model
model.fit(X_train, y_train, epochs=200, batch_size=8, verbose=0)
…
define model
model = Sequential()
model.add(Dense(10, activation=‘relu’, kernel_initializer=‘he_normal’, input_shape=(n_features,)))
model.add(Dense(8, activation=‘relu’, kernel_initializer=‘he_normal’))
model.add(Dense(1))
compile the model
model.compile(optimizer=‘adam’, loss=‘mse’)
fit the model
model.fit(X_train, y_train, epochs=200, batch_size=8, verbose=0)
The complete example is listed below.
k-fold cross-validation of deeper model for the auto insurance regression dataset
from numpy import mean
from numpy import std
from pandas import read_csv
from sklearn.model_selection import KFold
from sklearn.metrics import mean_absolute_error
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
from matplotlib import pyplot
load the dataset
path = ‘https://raw.githubusercontent.com/jbrownlee/Datasets/master/auto-insurance.csv’
df = read_csv(path, header=None)
split into input and output columns
X, y = df.values[:, :-1], df.values[:, -1]
prepare cross validation
kfold = KFold(10)
enumerate splits
scores = list()
for train_ix, test_ix in kfold.split(X, y):
# split data
X_train, X_test, y_train, y_test = X[train_ix], X[test_ix], y[train_ix], y[test_ix]
# determine the number of input features
n_features = X.shape[1]
# define model
model = Sequential()
model.add(Dense(10, activation=‘relu’, kernel_initializer=‘he_normal’, input_shape=(n_features,)))
model.add(Dense(8, activation=‘relu’, kernel_initializer=‘he_normal’))
model.add(Dense(1))
# compile the model
model.compile(optimizer=‘adam’, loss=‘mse’)
# fit the model
model.fit(X_train, y_train, epochs=200, batch_size=8, verbose=0)
# predict test set
yhat = model.predict(X_test)
# evaluate predictions
score = mean_absolute_error(y_test, yhat)
print(’>%.3f’ % score)
scores.append(score)
summarize all scores
print(‘Mean MAE: %.3f (%.3f)’ % (mean(scores), std(scores)))
k-fold cross-validation of deeper model for the auto insurance regression dataset
from numpy import mean
from numpy import std
from pandas import read_csv
from sklearn.model_selection import KFold
from sklearn.metrics import mean_absolute_error
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
from matplotlib import pyplot
load the dataset
path = ‘https://raw.githubusercontent.com/jbrownlee/Datasets/master/auto-insurance.csv’
df = read_csv(path, header=None)
split into input and output columns
X, y = df.values[:, :-1], df.values[:, -1]
prepare cross validation
kfold = KFold(10)
enumerate splits
scores = list()
for train_ix, test_ix in kfold.split(X, y):
# split data
X_train, X_test, y_train, y_test = X[train_ix], X[test_ix], y[train_ix], y[test_ix]
# determine the number of input features
n_features = X.shape[1]
# define model
model = Sequential()
model.add(Dense(10, activation=‘relu’, kernel_initializer=‘he_normal’, input_shape=(n_features,)))
model.add(Dense(8, activation=‘relu’, kernel_initializer=‘he_normal’))
model.add(Dense(1))
# compile the model
model.compile(optimizer=‘adam’, loss=‘mse’)
# fit the model
model.fit(X_train, y_train, epochs=200, batch_size=8, verbose=0)
# predict test set
yhat = model.predict(X_test)
# evaluate predictions
score = mean_absolute_error(y_test, yhat)
print(’>%.3f’ % score)
scores.append(score)
summarize all scores
print(‘Mean MAE: %.3f (%.3f)’ % (mean(scores), std(scores)))
Running reports the mean and standard deviation of the MAE at the end of the run.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
In this case, we can see that the MLP model achieved a MAE of about 35.384, which is slightly better than the baseline model that achieved an MAE of about 38.913.
Mean MAE: 35.384 (14.951)
1
Mean MAE: 35.384 (14.951)
Next, let’s try using the same model with a power transform for the input and target variables as we did in the previous section.
The complete example is listed below.
k-fold cross-validation of deeper model with data transforms
from numpy import mean
from numpy import std
from pandas import read_csv
from sklearn.model_selection import KFold
from sklearn.metrics import mean_absolute_error
from sklearn.preprocessing import PowerTransformer
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
from matplotlib import pyplot
load the dataset
path = ‘https://raw.githubusercontent.com/jbrownlee/Datasets/master/auto-insurance.csv’
df = read_csv(path, header=None)
split into input and output columns
X, y = df.values[:, :-1], df.values[:, -1]
prepare cross validation
kfold = KFold(10)
enumerate splits
scores = list()
for train_ix, test_ix in kfold.split(X, y):
# split data
X_train, X_test, y_train, y_test = X[train_ix], X[test_ix], y[train_ix], y[test_ix]
# ensure target is a 2d array
y_train, y_test = y_train.reshape((len(y_train),1)), y_test.reshape((len(y_test),1))
# prepare input data
pt1 = PowerTransformer()
pt1.fit(X_train)
X_train = pt1.transform(X_train)
X_test = pt1.transform(X_test)
# prepare target
pt2 = PowerTransformer()
pt2.fit(y_train)
y_train = pt2.transform(y_train)
y_test = pt2.transform(y_test)
# determine the number of input features
n_features = X.shape[1]
# define model
model = Sequential()
model.add(Dense(10, activation=‘relu’, kernel_initializer=‘he_normal’, input_shape=(n_features,)))
model.add(Dense(8, activation=‘relu’, kernel_initializer=‘he_normal’))
model.add(Dense(1))
# compile the model
model.compile(optimizer=‘adam’, loss=‘mse’)
# fit the model
model.fit(X_train, y_train, epochs=200, batch_size=8, verbose=0)
# predict test set
yhat = model.predict(X_test)
# inverse transforms
y_test = pt2.inverse_transform(y_test)
yhat = pt2.inverse_transform(yhat)
# evaluate predictions
score = mean_absolute_error(y_test, yhat)
print(’>%.3f’ % score)
scores.append(score)
summarize all scores
print(‘Mean MAE: %.3f (%.3f)’ % (mean(scores), std(scores)))
k-fold cross-validation of deeper model with data transforms
from numpy import mean
from numpy import std
from pandas import read_csv
from sklearn.model_selection import KFold
from sklearn.metrics import mean_absolute_error
from sklearn.preprocessing import PowerTransformer
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
from matplotlib import pyplot
load the dataset
path = ‘https://raw.githubusercontent.com/jbrownlee/Datasets/master/auto-insurance.csv’
df = read_csv(path, header=None)
split into input and output columns
X, y = df.values[:, :-1], df.values[:, -1]
prepare cross validation
kfold = KFold(10)
enumerate splits
scores = list()
for train_ix, test_ix in kfold.split(X, y):
# split data
X_train, X_test, y_train, y_test = X[train_ix], X[test_ix], y[train_ix], y[test_ix]
# ensure target is a 2d array
y_train, y_test = y_train.reshape((len(y_train),1)), y_test.reshape((len(y_test),1))
# prepare input data
pt1 = PowerTransformer()
pt1.fit(X_train)
X_train = pt1.transform(X_train)
X_test = pt1.transform(X_test)
# prepare target
pt2 = PowerTransformer()
pt2.fit(y_train)
y_train = pt2.transform(y_train)
y_test = pt2.transform(y_test)
# determine the number of input features
n_features = X.shape[1]
# define model
model = Sequential()
model.add(Dense(10, activation=‘relu’, kernel_initializer=‘he_normal’, input_shape=(n_features,)))
model.add(Dense(8, activation=‘relu’, kernel_initializer=‘he_normal’))
model.add(Dense(1))
# compile the model
model.compile(optimizer=‘adam’, loss=‘mse’)
# fit the model
model.fit(X_train, y_train, epochs=200, batch_size=8, verbose=0)
# predict test set
yhat = model.predict(X_test)
# inverse transforms
y_test = pt2.inverse_transform(y_test)
yhat = pt2.inverse_transform(yhat)
# evaluate predictions
score = mean_absolute_error(y_test, yhat)
print(’>%.3f’ % score)
scores.append(score)
summarize all scores
print(‘Mean MAE: %.3f (%.3f)’ % (mean(scores), std(scores)))
Running reports the mean and standard deviation of the MAE at the end of the run.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
In this case, we can see that the MLP model achieved a MAE of about 37.371, which is better than the baseline model, but not better than the deeper baseline model.
Perhaps this transform is not as helpful as we initially thought.
Mean MAE: 37.371 (29.326)
1
Mean MAE: 37.371 (29.326)
An alternate transform is to normalize the input and target variables.
This means to scale the values of each variable to the range [0, 1]. We can achieve this using the MinMaxScaler; for example:
…
prepare input data
pt1 = MinMaxScaler()
pt1.fit(X_train)
X_train = pt1.transform(X_train)
X_test = pt1.transform(X_test)
prepare target
pt2 = MinMaxScaler()
pt2.fit(y_train)
y_train = pt2.transform(y_train)
y_test = pt2.transform(y_test)
…
prepare input data
pt1 = MinMaxScaler()
pt1.fit(X_train)
X_train = pt1.transform(X_train)
X_test = pt1.transform(X_test)
prepare target
pt2 = MinMaxScaler()
pt2.fit(y_train)
y_train = pt2.transform(y_train)
y_test = pt2.transform(y_test)
Tying this together, the complete example of evaluating the deeper MLP with data normalization is listed below.
k-fold cross-validation of deeper model with normalization transforms
from numpy import mean
from numpy import std
from pandas import read_csv
from sklearn.model_selection import KFold
from sklearn.metrics import mean_absolute_error
from sklearn.preprocessing import MinMaxScaler
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
from matplotlib import pyplot
load the dataset
path = ‘https://raw.githubusercontent.com/jbrownlee/Datasets/master/auto-insurance.csv’
df = read_csv(path, header=None)
split into input and output columns
X, y = df.values[:, :-1], df.values[:, -1]
prepare cross validation
kfold = KFold(10)
enumerate splits
scores = list()
for train_ix, test_ix in kfold.split(X, y):
# split data
X_train, X_test, y_train, y_test = X[train_ix], X[test_ix], y[train_ix], y[test_ix]
# ensure target is a 2d array
y_train, y_test = y_train.reshape((len(y_train),1)), y_test.reshape((len(y_test),1))
# prepare input data
pt1 = MinMaxScaler()
pt1.fit(X_train)
X_train = pt1.transform(X_train)
X_test = pt1.transform(X_test)
# prepare target
pt2 = MinMaxScaler()
pt2.fit(y_train)
y_train = pt2.transform(y_train)
y_test = pt2.transform(y_test)
# determine the number of input features
n_features = X.shape[1]
# define model
model = Sequential()
model.add(Dense(10, activation=‘relu’, kernel_initializer=‘he_normal’, input_shape=(n_features,)))
model.add(Dense(8, activation=‘relu’, kernel_initializer=‘he_normal’))
model.add(Dense(1))
# compile the model
model.compile(optimizer=‘adam’, loss=‘mse’)
# fit the model
model.fit(X_train, y_train, epochs=200, batch_size=8, verbose=0)
# predict test set
yhat = model.predict(X_test)
# inverse transforms
y_test = pt2.inverse_transform(y_test)
yhat = pt2.inverse_transform(yhat)
# evaluate predictions
score = mean_absolute_error(y_test, yhat)
print(’>%.3f’ % score)
scores.append(score)
summarize all scores
print(‘Mean MAE: %.3f (%.3f)’ % (mean(scores), std(scores)))
k-fold cross-validation of deeper model with normalization transforms
from numpy import mean
from numpy import std
from pandas import read_csv
from sklearn.model_selection import KFold
from sklearn.metrics import mean_absolute_error
from sklearn.preprocessing import MinMaxScaler
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
from matplotlib import pyplot
load the dataset
path = ‘https://raw.githubusercontent.com/jbrownlee/Datasets/master/auto-insurance.csv’
df = read_csv(path, header=None)
split into input and output columns
X, y = df.values[:, :-1], df.values[:, -1]
prepare cross validation
kfold = KFold(10)
enumerate splits
scores = list()
for train_ix, test_ix in kfold.split(X, y):
# split data
X_train, X_test, y_train, y_test = X[train_ix], X[test_ix], y[train_ix], y[test_ix]
# ensure target is a 2d array
y_train, y_test = y_train.reshape((len(y_train),1)), y_test.reshape((len(y_test),1))
# prepare input data
pt1 = MinMaxScaler()
pt1.fit(X_train)
X_train = pt1.transform(X_train)
X_test = pt1.transform(X_test)
# prepare target
pt2 = MinMaxScaler()
pt2.fit(y_train)
y_train = pt2.transform(y_train)
y_test = pt2.transform(y_test)
# determine the number of input features
n_features = X.shape[1]
# define model
model = Sequential()
model.add(Dense(10, activation=‘relu’, kernel_initializer=‘he_normal’, input_shape=(n_features,)))
model.add(Dense(8, activation=‘relu’, kernel_initializer=‘he_normal’))
model.add(Dense(1))
# compile the model
model.compile(optimizer=‘adam’, loss=‘mse’)
# fit the model
model.fit(X_train, y_train, epochs=200, batch_size=8, verbose=0)
# predict test set
yhat = model.predict(X_test)
# inverse transforms
y_test = pt2.inverse_transform(y_test)
yhat = pt2.inverse_transform(yhat)
# evaluate predictions
score = mean_absolute_error(y_test, yhat)
print(’>%.3f’ % score)
scores.append(score)
summarize all scores
print(‘Mean MAE: %.3f (%.3f)’ % (mean(scores), std(scores)))
Running reports the mean and standard deviation of the MAE at the end of the run.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
In this case, we can see that the MLP model achieved a MAE of about 30.388, which is better than any other configuration we have tried so far.
Mean MAE: 30.388 (14.258)
1
Mean MAE: 30.388 (14.258)
We could continue to test alternate configurations to the model architecture (more or fewer nodes or layers), learning hyperparameters (more or fewer batches), and data transforms.