What are Weighted Average Ensemble?

Data Science, AI and ML
#1

A weighted average might be considered one step above a voting ensemble.

Like stacking and voting ensembles, a weighted average uses a diverse collection of model types as contributing members.

Unlike voting, a weighted average assumes that some contributing members are better than others and weighs contributions from models accordingly.

The simplest weighted average ensemble weighs each model based on its performance on a training dataset. An improvement over this naive approach is to weigh each member based on its performance on a hold-out dataset, such as a validation set or out-of-fold predictions during k-fold cross-validation.

One step further might involve tuning the coefficient weightings for each model using an optimization algorithm and performance on a holdout dataset.

These continued improvements of a weighted average model begin to resemble a primitive stacking model with a linear model trained to combine the predictions.

  • Member Assessment: Member performance on training dataset.
  • Combine With Model: Weighted average of predictions.