1. Machine Learning Algorithms Expect Numbers
Even though your data is represented in one large table of rows and columns, the variables in the table may have different data types.
Some variables may be numeric, such as integers, floating-point values, ranks, rates, percentages, and so on. Other variables may be names, categories, or labels represented with characters or words, and some may be binary, represented with 0 and 1 or True and False.
The problem is, machine learning algorithms at their core operate on numeric data. They take numbers as input and predict a number as output. All data is seen as vectors and matrices, using the terminology from linear algebra.
As such, raw data must be changed prior to training, evaluating, and using machine learning models.
Sometimes the changes to the data can be managed internally by the machine learning algorithm; most commonly, this must be handled by the machine learning practitioner prior to modeling in what is commonly referred to as “ data preparation ” or “ data pre-processing “.
2. Machine Learning Algorithms Have Requirements
Even if your raw data contains only numbers, some data preparation is likely required.
There are many different machine learning algorithms to choose from for a given predictive modeling project. We cannot know which algorithm will be appropriate, let alone the most appropriate for our task. Therefore, it is a good practice to evaluate a suite of different candidate algorithms systematically and discover what works well or best on our data.
The problem is, each algorithm has specific requirements or expectations with regard to the data.
… data preparation can make or break a model’s predictive ability. Different models have different sensitivities to the type of predictors in the model; how the predictors enter the model is also important.
— Page 27, Applied Predictive Modeling, 2013.
For example, some algorithms assume each input variable, and perhaps the target variable, to have a specific probability distribution. This is often the case for linear machine learning models that expect each numeric input variable to have a Gaussian probability distribution.
This means that if you have input variables that are not Gaussian or nearly Gaussian, you might need to change them so that they are Gaussian or more Gaussian. Alternatively, it may encourage you to reconfigure the algorithm to have a different expectation on the data.
Some algorithms are known to perform worse if there are input variables that are irrelevant or redundant to the target variable. There are also algorithms that are negatively impacted if two or more input variables are highly correlated. In these cases, irrelevant or highly correlated variables may need to be identified and removed, or alternate algorithms may need to be used.
There are also algorithms that have very few requirements about the probability distribution of input variables or the presence of redundancies, but in turn, may require many more examples (rows) in order to learn how to make good predictions.
The need for data pre-processing is determined by the type of model being used. Some procedures, such as tree-based models, are notably insensitive to the characteristics of the predictor data. Others, like linear regression, are not.
— Page 27, Applied Predictive Modeling, 2013.
As such, there is an interplay between the data and the choice of algorithms. Primarily, the algorithms impose expectations on the data, and adherence to these expectations requires the data to be appropriately prepared. Conversely, the form of the data may help choose algorithms to evaluate that are more likely to be effective.
3. Model Performance Depends on Data
Even if you prepare your data to meet the expectations of each model, you may not get the best performance.
Often, the performance of machine learning algorithms that have strong expectations degrades gracefully to the degree that the expectation is violated.
Further, it is common for an algorithm to perform well or better than other methods, even when its expectations have been ignored or completely violated. It is a common enough situation that this must be factored into the preparation and evaluation of machine learning algorithms.
The idea that there are different ways to represent predictors in a model, and that some of these representations are better than others, leads to the idea of feature engineering — the process of creating representations of data that increase the effectiveness of a model.
— Page 3, Feature Engineering and Selection, 2019.
The performance of a machine learning algorithm is only as good as the data used to train it. This is often summarized as “ garbage in, garbage out “. Garbage is harsh, but it could mean a “ weak representation ” of the problem that insufficiently captures the dynamics required to learn how to map examples of inputs to outputs.
Let’s take for granted that we have “ sufficient ” data to capture the relationship between input and output variables. It’s a slippery and domain-specific principle, and in practice, we have the data that we have, and our job is to do the best we can with that data.
A dataset may be a “ weak representation ” of the problem we are trying to solve for many reasons, although there are two main classes of reason. It may be because complex nonlinear relationships are compressed in the raw data that can be unpacked using data preparation techniques. It may also be because the data is not perfect, ranging from mild random fluctuations in the observations, referred to as a statistical noise, to errors that result in out-of-range values and conflicting data.
- Complex Data : Raw data contains compressed complex nonlinear relationships that may need to be exposed
- Messy Data : Raw data contains statistical noise, errors, missing values, and conflicting examples.
We can think about getting the most out of our predictive modeling project in two ways: focus on the model and focus on the data.
We could minimally prepare the raw data and begin modeling. This puts full onus on the model to tease out the relationships in the data and learn the mapping function from inputs to outputs as best it can. This may be a reasonable path through a project and may require a large dataset and a flexible and powerful machine learning algorithm with few expectations, such as random forest or gradient boosting.
Alternately, we could push the onus back onto the data and the data preparation process. This requires that each row of data maximumly or best expresses the information content of the data for modeling. Just like denormalization of data in a relational database to rows and columns, data preparation can denormalize the complex structure inherent in each single observation. This is also a reasonable path. It may require more knowledge of the data than is available but allows good or even best modeling performance to be achieved almost irrespective of the machine learning algorithm used.
Often a balance between these approaches is pursued on any given project. That is both exploring powerful and flexible machine learning algorithms and using data preparation to best expose the structure of the data to the learning algorithms.
This is all to say, data preprocessing is a path to better data, and in turn, better model performance.