Overfitting in Machine Learning, How to overcome from this situation?

Data Science, AI and ML
,
#1

[Overfitting] refers to a model that models the training data too well.

Overfitting happens when a model learns the detail and noise in the training data to the extent that it negatively impacts the performance of the model on new data. This means that the noise or random fluctuations in the training data is picked up and learned as concepts by the model. The problem is that these concepts do not apply to new data and negatively impact the models ability to generalize.
Overfitting is the situation of - Low bias and High Variance.

Overfitting is more likely with nonparametric and nonlinear models that have more flexibility when learning a target function. As such, many nonparametric machine learning algorithms also include parameters or techniques to limit and constrain how much detail the model learns.

Overfitting is such a problem because the evaluation of machine learning algorithms on training data is different from the evaluation we actually care the most about, namely how well the algorithm performs on unseen data.

There are two important techniques that you can use when evaluating machine learning algorithms to limit overfitting:

1. Use a resampling technique to estimate model accuracy.
2. Hold back a validation dataset.

The most popular resampling technique is k-fold cross validation. It allows you to train and test your model k-times on different subsets of training data and build up an estimate of the performance of a machine learning model on unseen data.

A validation dataset is simply a subset of your training data that you hold back from your machine learning algorithms until the very end of your project. After you have selected and tuned your machine learning algorithms on your training dataset you can evaluate the learned models on the validation dataset to get a final objective idea of how the models might perform on unseen data.

Using cross validation is a gold standard in applied machine learning for estimating model accuracy on unseen data. If you have the data, using a validation dataset is also an excellent practice.

#2

Overfitting is a modeling error which occurs when a function is too closely fit a limited set of data points. Overfitting the model generally takes the form of making an overly complex model to explain idiosyncrasies in the data under study.

The polynomial which represents the equations of this model is very complex because it tries to capture every data point perfectly.

You don’t correct overfitting, but you prevent it. Below are some of the things to do in order to prevent overfitting:

  1. Use regularization methods (l1, l2, dropout, etc.)
  2. Keep complexity of your model at a reasonable level.
  3. Use more data. If applicable, use augmentation.
  4. If the training algorithm is iterative (such as gradient descent), monitor the performance in a validation set and use early stopping when this performance starts to drop (or becomes stable).