73
$\begingroup$

I have a four layer CNN to predict response to cancer using MRI data. I use ReLU activations to introduce nonlinearities. The train accuracy and loss monotonically increase and decrease respectively. But, my test accuracy starts to fluctuate wildly. I have tried changing the learning rate, reduce the number of layers. But, it doesn't stop the fluctuations. I even read this answer and tried following the directions in that answer, but not luck again. Could anyone help me figure out where I am going wrong?

Screenshot

$\endgroup$
7
  • $\begingroup$ stats.stackexchange.com/questions/189774/… $\endgroup$ Jan 8, 2017 at 6:51
  • $\begingroup$ Yes, I read that answer. Shuffling the validation data did not help $\endgroup$ Jan 8, 2017 at 6:52
  • 7
    $\begingroup$ Because you haven't shared your code snippet, hence I can't say much what's wrong in your architecture. But in your screen shot, seeing your training and validation accuracy, it's crystal clear that your network is overfitting. It would be better if you share your code snippet here . $\endgroup$
    – enterML
    Mar 1, 2017 at 11:54
  • $\begingroup$ how many samples do you have? maybe the fluctuation is not really signifficant. Also, the accuracy is horrible measure $\endgroup$
    – rep_ho
    Feb 2, 2018 at 11:18
  • 1
    $\begingroup$ i suggest to use batch normalization and drop out in the architecture of network $\endgroup$ Aug 29, 2020 at 23:26

7 Answers 7

51
$\begingroup$

If I understand the definition of accuracy correctly, accuracy (% of data points classified correctly) is less cumulative than let's say MSE (mean squared error). That's why you see that your loss is rapidly increasing, while accuracy is fluctuating.

Intuitively, this basically means, that some portion of examples is classified randomly, which produces fluctuations, as the number of correct random guesses always fluctuate (imagine accuracy when coin should always return "heads"). Basically sensitivity to noise (when classification produces random result) is a common definition of overfitting (see wikipedia):

In statistics and machine learning, one of the most common tasks is to fit a "model" to a set of training data, so as to be able to make reliable predictions on general untrained data. In overfitting, a statistical model describes random error or noise instead of the underlying relationship

Another evidence of overfitting is that your loss is increasing, Loss is measured more precisely, it's more sensitive to the noisy prediction if it's not squashed by sigmoids/thresholds (which seems to be your case for the Loss itself). Intuitively, you can imagine a situation when network is too sure about output (when it's wrong), so it gives a value far away from threshold in case of random misclassification.

Regarding your case, your model is not properly regularized, possible reasons:

  • not enough data-points, too much capacity
  • ordering
  • no/wrong feature scaling/normalization
  • learning rate: $\alpha$ is too large, so SGD jumps too far and misses the area near local minima. This would be extreme case of "under-fitting" (insensitivity to data itself), but might generate (kind of) "low-frequency" noise on the output by scrambling data from the input - contrary to the overfitting intuition, it would be like always guessing heads when predicting a coin. As @JanKukacka pointed out, arriving at the area "too close to" a minima might cause overfitting, so if $\alpha$ is too small it would get sensitive to "high-frequency" noise in your data. $\alpha$ should be somewhere in between.

Possible solutions:

  • obtain more data-points (or artificially expand the set of existing ones)
  • play with hyper-parameters (increase/decrease capacity or regularization term for instance)
  • regularization: try dropout, early-stopping, so on
$\endgroup$
6
  • $\begingroup$ Regarding: "Loss is measured more precisely, it's more sensitive to the noisy prediction because it's not squashed by sigmoids/thresholds", I agree with no thresholding, but if you are using e.g. binary cross entropy as your loss function, the sigmoid still plays a role. $\endgroup$
    – Zhubarb
    Feb 2, 2018 at 10:02
  • 1
    $\begingroup$ Regarding learning rate and sgd missing the minima: reaching the minimum would most likely mean overfitting (because it is the minimum on the training set) $\endgroup$ Feb 2, 2018 at 10:54
  • $\begingroup$ @Berkmeister true, I've rephrased a bit (see edit). My thinking there was that increased Loss is a sign of non-squashed function being used. $\endgroup$
    – dk14
    Feb 2, 2018 at 10:54
  • $\begingroup$ @JanKukacka you mean global minima? I implied the local minima (actually near local minima) - in the meaning that if it's too far from any minima, it would be under-fitting then. Probably, I should describe it more carefully (see edit), thanks. $\endgroup$
    – dk14
    Feb 2, 2018 at 10:59
  • $\begingroup$ @dk14 I assume global minimum cannot be in practice reached, so I mean rather local minima. If you are too far, you might be under-fitting, but if you are too close, you are most likely overfitting. There is an interesting work by Moritz Hardt "Train faster, generalize better: Stability of stochastic gradient descent" (arxiv.org/abs/1509.01240) putting bounds on the relation between training and testing error when training with SGD. $\endgroup$ Feb 2, 2018 at 11:12
18
$\begingroup$

This question is old but posting this as it hasn't been pointed out yet:

Possibility 1: You're applying some sort of preprocessing (zero meaning, normalizing, etc.) to either your training set or validation set, but not the other.

Possibility 2: If you built some layers that perform differently during training and inference from scratch, your model might be incorrectly implemented (e.g. are moving mean and moving standard deviation for batch normalization getting updated during training? If using dropout, are weights scaled properly during inference?). This might be the case if your code implements these things from scratch and does not use Tensorflow/Pytorch's builtin functions.

Possibility 3: Overfitting, as everybody has pointed out. I find the other two options more likely in your specific situation as your validation accuracy is stuck at 50% from epoch 3. Generally, I would be more concerned about overfitting if this was happening in a later stage (unless you have a very specific problem at hand).

$\endgroup$
4
  • $\begingroup$ I am having a problem that is kind of similar but not completely, more details here: stackoverflow.com/questions/55348052/… In my case, I do actually have a consistent high accuracy with test data and during training, the validation "accuracy" (not loss) is higher than the training accuracy. But the fact that it never converges and oscillates makes me think of overfitting, while some suggest that is not the case, so I wonder if it is and what is the justification if it is not. $\endgroup$
    – dusa
    Mar 26, 2019 at 16:29
  • 1
    $\begingroup$ This is by far the most plausible explanation of the answers given. Notice that high batch normalization momentum (eg. 0.999, or even the Keras default 0.99) in combination with a high learning rate can also produce very different behavior in training and evaluation as layer statistics lag very far behind. In that case reducing momentum to something like 0.9 should do the trick. I have had a similar problem as OP and this did the trick. $\endgroup$
    – kristjan
    Aug 19, 2019 at 11:49
  • $\begingroup$ Thank you for this answer! I double checked if dropout is working correctly in my model. It turned out that using Keras' functional API, training=True is not set automatically! This stands in contrast to the documentation: Dropout. $\endgroup$
    – Domi W
    Jul 22, 2020 at 15:44
  • 3
    $\begingroup$ One way to test for Possibility 1 would be to use training data for validation. If you still see the issue, then Possibility 1 could be the case. $\endgroup$
    – Konstantin
    Mar 12, 2021 at 12:16
13
$\begingroup$

Adding to the answer by @dk14 . If you are still seeing fluctuations after properly regularising your model, these could be the possible reasons:

  • Using a random sample from your validation set: It means your validation set at each evaluation step is different, so is your validation-loss.
  • Using a weighted loss-function(which is used in case of highly imbalanced class-problems). At train step, you weigh your loss function based on class-weights, while at dev step you just calculate the un-weighted loss. In such case, though your network is stepping into convergence, you might see lots of fluctuations in validation loss after each train-step. But if you wait for a bigger picture, you can see that your network is actually converging to a minima with fluctuations wearing out.(see the attached images for one such example). enter image description hereenter image description here
$\endgroup$
6
$\begingroup$

Your validation accuracy on a binary classification problem (I assume) is "fluctuating" around 50%, that means your model is giving completely random predictions (sometimes it guesses correctly few samples more, sometimes a few samples less). Generally, your model is not better than flipping a coin.

The reason the validation loss is more stable is that it is a continuous function: It can distinguish that prediction 0.9 for a positive sample is more correct than a prediction 0.51. For accuracy, you round these continuous logit predictions to $\{0;1\}$ and simply compute the percentage of correct predictions. Now, since your model is guessing, it is most likely predicting values near 0.5 for all samples, let's say a sample gets 0.49 after one epoch and 0.51 in the next. From the loss perspective the incorrectness of the prediction did not change much, whereas the accuracy is sensitive even to these small differences.

Anyway, as others have already pointed out, your model is experiencing severe overfitting. My guess is that your problem is too complicated, i.e. it is very difficult to extract the desired information from your data, and such simple end2end trained 4-layer conv-net has no chance of learning it.

$\endgroup$
3
$\begingroup$

Definitely over-fitting. The gap between accuracy on training data and test data shows you have over fitted on training. Maybe regularization can help.

$\endgroup$
1
$\begingroup$

There are few ways to try in your situation. Firstly try to increase the batch size, which helps the mini-batch SGD less wandering wildly. Secondly tuning the learning rate, probably set it smaller. Thirdly, try different optimizer, for instance Adam or RMSProp which are able to adapt learning rates for wrt features. If possible try augmenting your data. Lastly, try Bayesian neural networks via dropout approximation, a very interesting work of Yarin Gal https://arxiv.org/abs/1506.02158

$\endgroup$
1
$\begingroup$

Have you tried a smaller network? Considering your training accuracy can reach >.99, your network seems have enough connections to fully model your data, but you may have extraneous connections that are learning randomly (i.e. overfitting).

In my experience, I've gotten the holdout validation accuracy to stabilize with a smaller network by trying various networks such as ResNet, VGG, and even simpler networks.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.