What Makes a Good Ensemble

Data Science, AI and ML
#1

An ensemble is a machine learning model that combines the predictions from multiple other models.

This often has the effect of reducing prediction error and improving the generalization of the model. But this is not always the case.

Sometimes the ensemble performs no better than a well-performing contributing member to the ensemble. Even worse, sometimes an ensemble will perform worse than any contributing member.

This raises the question as to what makes a good ensemble.

A good ensemble is an ensemble that performs better than any contributing member. That is, it is a model that has lower prediction error for regression or higher accuracy for classification.

  • Good Ensemble : A model that performs better than any single contributing model.

This can be evaluated empirically using a train and test set or a resampling technique like k-fold cross-validation. The results can similarly be estimated for each contributing model and the results compared directly to see if the definition of a “ good ensemble ” is met. This is a widely studied question with many ideas. The consistency is that a good ensemble has diversity .