I have a network that I am training for an image classification task, with about 30 layers. The network is softmax activated at the end and uses cross entropy loss.

When I train, I have been noticing the following:

  1. The training goes well for a few epochs. The losses keep decreasing and the accuracies increase. The probability values given by the softmax are high for one class (may be different each time) over the others, and it is usually the correct class. As the data type can hold limited amount of information, sometimes it is equal to 1 for a given class.
  2. After some point, the probability is 1 for an incorrect class. This leads the loss to be infinite. To correct this, I have clipped all values given as outputs to be close to, but not equal, to 0 or 1 whenever appropriate. This results in a large loss as compared to previous iterations, an order of magnitude more.
  3. Soon (may not be immediate) after this happens, the network starts classifying every input with probability 1 to some class.
  4. After a few more iterations, a particular class "takes over", that is, the network classifies every input with probability 1 to be in that particular class. When point 3. and this happen, the loss is about 10x as compared to before, when the training was proceeding normally. Yet, this loss does not decrease any further, the reason which I found is mentioned in the next point.
  5. On investigating, I realised that by the time step 4 sets in (or a bit before), the gradients have decreased to an extent that very soon, starting from the final layer, the gradient is zero. So, no updates are ever made after this and any further training is not useful.

A few points:

  1. I have used leaky ReLUs with a leakiness of $\frac{1}{3}$ to activate all my dense layers. I have read about the dying ReLU problem, but as I understand it, leaky ReLUs should mitigate that.
  2. My weights are initialized with an Orthogonal initializer.
  3. The stage at which step 3 starts vastly varies across different training attempts.
  4. My classes are imbalanced, and to help correct any bias, I am using oversampling.

My questions are:

  1. Why does the gradient become $0$, especially when I am using leaky ReLUs? How can I correct this?
  2. Is there something wrong in the fact that my network tends to classify inputs occasionally with very high (1 for the datatype that stores it) probability to belong to a particular class?

Thank you.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.