Mean Squared Error
Mean Squared Error, or MSE for short, is a popular error metric for regression problems.
It is also an important loss function for algorithms fit or optimized using the least squares framing of a regression problem. Here “least squares” refers to minimizing the mean squared error between predictions and expected values.
The MSE is calculated as the mean or average of the squared differences between predicted and expected target values in a dataset.
MSE = 1 / N * sum for i to N (y_i – yhat_i)^2
Where y_i is the i’th expected value in the dataset and yhat_i is the i’th predicted value. The difference between these two values is squared, which has the effect of removing the sign, resulting in a positive error value.
The squaring also has the effect of inflating or magnifying large errors. That is, the larger the difference between the predicted and expected values, the larger the resulting squared positive error. This has the effect of “punishing” models more for larger errors when MSE is used as a loss function. It also has the effect of “punishing” models by inflating the average error score when used as a metric.
We can create a plot to get a feeling for how the change in prediction error impacts the squared error.
The example below gives a small contrived dataset of all 1.0 values and predictions that range from perfect (1.0) to wrong (0.0) by 0.1 increments. The squared error between each prediction and expected value is calculated and plotted to show the quadratic increase in squared error.
…
calculate error
err = (expected[i] - predicted[i])**2
The complete example is listed below.
example of increase in mean squared error
from matplotlib import pyplot
from sklearn.metrics import mean_squared_error
real value
expected = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
predicted value
predicted = [1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1, 0.0]
calculate errors
errors = list()
for i in range(len(expected)):
# calculate error
err = (expected[i] - predicted[i])**2
# store error
errors.append(err)
# report error
print(‘>%.1f, %.1f = %.3f’ % (expected[i], predicted[i], err))
plot errors
pyplot.plot(errors)
pyplot.xticks(ticks=[i for i in range(len(errors))], labels=predicted)
pyplot.xlabel(‘Predicted Value’)
pyplot.ylabel(‘Mean Squared Error’)
pyplot.show()
Running the example first reports the expected value, predicted value, and squared error for each case.
We can see that the error rises quickly, faster than linear (a straight line).
1.0, 1.0 = 0.000
1.0, 0.9 = 0.010
1.0, 0.8 = 0.040
1.0, 0.7 = 0.090
1.0, 0.6 = 0.160
1.0, 0.5 = 0.250
1.0, 0.4 = 0.360
1.0, 0.3 = 0.490
1.0, 0.2 = 0.640
1.0, 0.1 = 0.810
1.0, 0.0 = 1.000
A line plot is created showing the curved or super-linear increase in the squared error value as the difference between the expected and predicted value is increased.
The curve is not a straight line as we might naively assume for an error metric.
The individual error terms are averaged so that we can report the performance of a model with regard to how much error the model makes generally when making predictions, rather than specifically for a given example.
The units of the MSE are squared units.
For example, if your target value represents “dollars,” then the MSE will be “squared dollars.” This can be confusing for stakeholders; therefore, when reporting results, often the root mean squared error is used instead (discussed in the next section).
The mean squared error between your expected and predicted values can be calculated using the mean_squared_error() function from the scikit-learn library.
The function takes a one-dimensional array or list of expected values and predicted values and returns the mean squared error value.
…
calculate errors
errors = mean_squared_error(expected, predicted)
The example below gives an example of calculating the mean squared error between a list of contrived expected and predicted values.
example of calculate the mean squared error
from sklearn.metrics import mean_squared_error
real value
expected = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
predicted value
predicted = [1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1, 0.0]
calculate errors
errors = mean_squared_error(expected, predicted)
report error
print(errors)
Running the example calculates and prints the mean squared error.
0.35000000000000003
A perfect mean squared error value is 0.0, which means that all predictions matched the expected values exactly.
This is almost never the case, and if it happens, it suggests your predictive modeling problem is trivial.
A good MSE is relative to your specific dataset.
It is a good idea to first establish a baseline MSE for your dataset using a naive predictive model, such as predicting the mean target value from the training dataset. A model that achieves an MSE better than the MSE for the naive model has skill.