What Is Ensemble Diversity?

Data Science, AI and ML
#1

Ensemble diversity refers to differences in the decisions or predictions made by the ensemble members. Two ensemble members that make identical predictions are considered not diverse.

Ensembles that make completely different predictions in all cases are maximally diverse, although this is most unlikely. Therefore, some level of diversity in predictions is desired or even required in order to construct a good ensemble. This is often simplified in discussion to the desire for diverse ensemble members, e.g. the models themselves, that in turn will produce diverse predictions, although diversity in predictions is truly what we seek.

Ideally, diversity would mean that the predictions made by each ensemble member are independent and uncorrelated. Independence is a term from probability theory and refers to the case where one event does not influence the probability of the occurrence of another event. Events can influence each other in many different ways. One ensemble may influence another if it attempts to correct the predictions made by it.

As such, depending on the ensemble type, models may be naturally dependent or independent.

  • Independence : Whether the occurrence of an event affects the probability of subsequent events.

Correlation is a term from statistics and refers to two variables changing together. It is common to calculate a normalized correlation score between -1.0 and 1.0 where a score of 0.0 indicates no correlation, e.g. uncorrelated. A score of 1.0 or -1.0 one indicates perfect positive and negative correlation respectively and indicates two models that always predict the same (or inverse) outcome. In practice, ensembles often exhibit a weak or modest positive correlation in their predictions.

  • Correlation : The degree to which variables change together.

As such, ensemble models are constructed from constituent models that may or may not have some dependency and some correlation between the decisions or predictions made. Constructing good ensembles is a challenging task.

For example, combining a bunch of top-performing models will likely result in a poor ensemble as the predictions made by the models will be highly correlated. Unintuitively, you might be better off combining the predictions from a few top-performing individual models with the prediction from a few weaker models. Sadly, there is no generally agreed upon measure of ensemble diversity and ideas of independence and correlation are only guides or proxies for thinking about the properties of good ensembles. Nevertheless, as guides, they are useful as we can devise techniques that attempt to reduce the correlation between the models.