I'm training a 2-layer CNN model on audio samples, represented as CQT. There are ≈160k samples, many that are very similar since they originate from the same instrument and/or audio file. 10% have been split out beforehand for validation. My question is, why does my validation loss go up, while the validation accuracy goes up as well. A typical example can be seen in the image below.
The model roughly looks like (conv/pool/relu)x2 -> flatten/dense -> dense/softmax. Categorical crossentopy as cost function.
The phenomena occurs both when validation split is randomly picked from training data, or picked from a completely different dataset. The only way I managed it to go in the "correct" direction (i.e. loss goes down, acc up) is when I use L2-regularization, or a global average pooling instead of the dense layers.