The objective for developing good ensembles is considered through the lens of increasing the diversity of ensemble members.
Models in an ensemble may be more dependent if they share the same algorithm used to train the model and/or the same dataset used to train the model. Models in an ensemble may have higher correlation between their predictions for the same general reasons.
Therefore, approaches that make the models and/or the training data more different are desirable. It can be helpful to have a framework for thinking about techniques for managing the diversity of ensembles, and there are many to choose from.
For example, Zhi-Hua Zhou in Chapter 5 of his 2012 book titled “Ensemble Methods: Foundations and Algorithms” proposes a framework of four approaches to generating diversity. In summary, they are:
- Data Sample Manipulation : e.g. sample the training dataset differently for each model.
- Input Feature Manipulation : e.g. train each model on different groups of input features.
- Learning Parameter Manipulation : e.g. train models with different hyperparameter values.
- Output Representation Manipulation : e.g. train models with differently modified target values.
Perhaps the most common approaches are changes to the training data, that is data samples and input features, often combined in a single approach.
For another example of a taxonomy for generating a diverse ensemble, Lior Rokach in Chapter 4 his 2010 book titled “Pattern Classification Using Ensemble Methods” suggest a similar taxonomy, summarized as:
-
Manipulating the Inducer , e.g. manipulating in how models are trained.
- Vary hyperparameters.
- Varying starting point.
- Vary optimization algorithm.
-
Manipulating the Training Sample , e.g. manipulating the data used for training.
- Resampling.
-
Changing the target attribute representation , e.g. manipulating the target variable.
- Vary encoding.
- Error-correcting codes.
- Label switching.
-
Partitioning the search space , e.g. manipulating the number of input features.
- Random subspace.
- Feature selection.
- Hybridization , e.g. varying model types or a mixture of the above methods.
The hybridization approach is common both in popular ensemble methods that combine multiple approaches at generating diversity in a single algorithm, as well as simply varying the algorithms (model types) that comprise the ensemble.This highlights that although we could investigate quantifying how diverse an ensemble is, that instead, most effort is on developing techniques that generate diversity.
We can harness this knowledge in developing our own ensemble methods, first trying standard and well-understood ensemble learning methods, then tailoring them for our specific dataset with model independence and correlation of predictions in mind in an effort to get the most out of them.