Combining Regression Predictions

Data Science, AI and ML
#1

Regression refers to predictive modeling problems that involve predicting a numeric value given an input.

The performance for a regression problem is often measured using average error, such as mean absolute error or root mean squared error.

Combining numerical predictions often involves using simple statistical methods; for example:

  • Mean Predicted Value
  • Median Predicted Value

Both give the central tendency of the distribution of predictions.

Averaging is the most popular and fundamental combination method for numeric outputs.

— Page 68, Ensemble Methods, 2012.

The mean, also called the average, is the normalized sum of the predictions. The Mean Predicted Value is more appropriate when the distribution of predictions is Gaussian or nearly Gaussian.

For example, the mean is calculated as the sum of predicted values divided by the total number of predictions. If three models predicted the following prices:

  • Model 1: 99.00
  • Model 2: 101.00
  • Model 3: 98.00

The mean predicted would be calculated as:

  • Mean Prediction = (99.00 + 101.00 + 98.00) / 3
  • Mean Prediction = 298.00 / 3
  • Mean Prediction = 99.33

Owing to its simplicity and effectiveness, simple averaging is among the most popularly used methods and represents the first choice in many real applications.

— Page 69, Ensemble Methods, 2012.

The median is the middle value if all predictions were ordered and is also referred to as the fifty-th percentile. The Median Predicted Value is more appropriate to use when the distribution of predictions is not known or does not follow a Gaussian probability distribution.

Depending on the nature of the prediction problem, a conservative prediction may be desired, such as the maximum or the minimum. Additionally, the distribution can be summarized to give a measure of uncertainty, such as reporting three values for each prediction:

  • Minimum Predicted Value
  • Median Predicted Value
  • Maximum Predicted Value

As with classification, the predictions made by each model can be weighted by expected model performance or some other value, and the weighted mean of the predictions can be reported.