What are exploding gradients?

Exploding gradients relate to the accumulation of significant error gradients, resulting in very large updates to the neural network model weights during training. This, in turn, leads to an unstable network.

The values of the weights can also become so large as to overflow and result in something called NaN values.