I'm training a 2-layer CNN model on audio samples, represented as CQT. There are ≈160k samples, many that are very similar since they originate from the same instrument and/or audio file. 10% have been split out beforehand for validation. My question is, why does my validation loss go up, while the validation accuracy goes up as well. A typical example can be seen in the image below.

Typical example

The model roughly looks like (conv/pool/relu)x2 -> flatten/dense -> dense/softmax. Categorical crossentopy as cost function.

The phenomena occurs both when validation split is randomly picked from training data, or picked from a completely different dataset. The only way I managed it to go in the "correct" direction (i.e. loss goes down, acc up) is when I use L2-regularization, or a global average pooling instead of the dense layers.

  • $\begingroup$ Interesting. I am also having the same phenomenon occur with my convolutional network. I will explore the L2 regularization that you suggested worked for you. One further thing that I noted for my network is that I get my lowest validation loss when I use dropout but my highest accuracy comes when I don't use it at all. $\endgroup$ Commented Jun 10, 2017 at 11:02
  • $\begingroup$ You can check my answer here: stats.stackexchange.com/questions/258166/… It gives some hints of why this kind of behavior can occur (although does not explain everything, I'm also looking for a better answer for this behavior ;) $\endgroup$
    – ahstat
    Commented Jun 10, 2017 at 11:10

1 Answer 1


There could be a lot going on here I'm going to give a layman's answer.

The accuracy is going to be how many correct observations out of all observations. If this is a classification problem and the classes are not balanced (120k : class1, 40k : class 2) then it is easily able to get a high accuracy by just picking class 1 more often. This is simple math 120k/160k.

The loss function can be more complex and can be more/less robust to class imbalances.

What I'm trying to say is that the functions for accuracy and loss aren't the same and thus can and will deviated depending on which once you are comparing. It will also depend on how well the classifier is doing for each class. It might be good to look at a confusion matrix if you don't have too many classes.

The scale isn't too extreme for the loss function either. The delta from the lowest to the highest loss is about 0.3. I don't have the experience to know if that is a large difference, but it might be something to consider.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.