Complete Guide to Prevent Overfitting in Neural Networks

Data Science, AI and ML
#1

Introduction

This is the final part of two-part Blog Series on Regularization Techniques for Neural Networks.

In the first part of the blog series, we discuss the basic concepts related to Underfitting and Overfitting and learn the following three methods to prevent overfitting in neural networks:

  • Reduce the Model Complexity
  • Data Augmentation
  • Weight Regularization

For part-1 of this series, refer to the link.

So, in continuation of the previous article, In this article we will cover the following techniques to prevent Overfitting in neural networks:

  • Dropout
  • Early Stopping
  • Weight Decay

Important Note

After the completion of each of the techniques, there is some practice (Test your Knowledge) questions given that you have to solve and give the answer in the comment box so that you can check your understanding of a particular technique.

Dropout

It is another regularization technique that prevents neural networks from overfitting. Regularization methods like L1 and L2 reduce overfitting by modifying the cost function but on the contrary, the Dropout technique modifies the network itself to prevent the network from overfitting.

Working Principle behind this Technique

It randomly drops some neurons except for the output layer from the neural network during training in each iteration or we can assign a probability p to all the neurons in a network so that they are temporarily ignored from calculations.

where p is known as the Dropout Rate and is usually instantiated to 0.5.

Then, as each iteration is going on, the neurons in each layer with the highest probability get dropped. This results in creating a smaller network with each pass on the training dataset(epoch). Since in each iteration, a random input value can be eliminated, the network tries to balance the risk and not to favor any of the features and reduces bias and noise.