Why is the validation accuracy fluctuating? I have a four layer CNN to predict response to cancer using MRI data. I use ReLU activations to introduce nonlinearities. The train accuracy and loss monotonically increase and decrease respectively. But, my test accuracy starts to fluctuate wildly. I have tried changing the learning rate, reduce the number of layers. But, it doesn't stop the fluctuations. I even read this answer and tried following the directions in that answer, but not luck again. Could anyone help me figure out where I am going wrong? 

 A: Your validation accuracy on a binary classification problem (I assume) is "fluctuating" around 50%, that means your model is giving completely random predictions (sometimes it guesses correctly few samples more, sometimes a few samples less). Generally, your model is not better than flipping a coin.
The reason the validation loss is more stable is that it is a continuous function: It can distinguish that prediction 0.9 for a positive sample is more correct than a prediction 0.51. For accuracy, you round these continuous logit predictions to $\{0;1\}$ and simply compute the percentage of correct predictions. Now, since your model is guessing, it is most likely predicting values near 0.5 for all samples, let's say a sample gets 0.49 after one epoch and 0.51 in the next. From the loss perspective the incorrectness of the prediction did not change much, whereas the accuracy is sensitive even to these small differences.
Anyway, as others have already pointed out, your model is experiencing severe overfitting. My guess is that your problem is too complicated, i.e. it is very difficult to extract the desired information from your data, and such simple end2end trained 4-layer conv-net has no chance of learning it.
A: If I understand the definition of accuracy correctly, accuracy (% of data points classified correctly) is less cumulative than let's say MSE (mean squared error). That's why you see that your loss is rapidly increasing, while accuracy is fluctuating. 
Intuitively, this basically means, that some portion of examples is classified randomly, which produces fluctuations, as the number of correct random guesses always fluctuate (imagine accuracy when coin should always return "heads"). Basically sensitivity to noise (when classification produces random result) is a common definition of overfitting (see wikipedia):

In statistics and machine learning, one of the most common tasks is to
  fit a "model" to a set of training data, so as to be able to make
  reliable predictions on general untrained data. In overfitting, a
  statistical model describes random error or noise instead of the
  underlying relationship

Another evidence of overfitting is that your loss is increasing, Loss is measured more precisely, it's more sensitive to the noisy prediction if it's not squashed by sigmoids/thresholds (which seems to be your case for the Loss itself). Intuitively, you can imagine a situation when network is too sure about output (when it's wrong), so it gives a value far away from threshold in case of random misclassification.
Regarding your case, your model is not properly regularized, possible reasons:


*

*not enough data-points, too much capacity

*ordering 

*no/wrong feature scaling/normalization

*learning rate: $\alpha$ is too large, so SGD jumps too far and misses the  area near local minima. This would be extreme case of "under-fitting" (insensitivity to data itself), but might generate (kind of) "low-frequency" noise on the output by scrambling data from the input - contrary to the overfitting intuition, it would be like always guessing heads when predicting a coin. As @JanKukacka pointed out, arriving at the area "too close to" a minima might cause overfitting, so if $\alpha$ is too small it would get sensitive to "high-frequency" noise in your data. $\alpha$ should be somewhere in between.


Possible solutions:


*

*obtain more data-points (or artificially expand the set of existing ones)

*play with hyper-parameters (increase/decrease capacity or regularization term for instance)

*regularization: try dropout, early-stopping, so on

A: Definitely over-fitting. The gap between accuracy on training data and test data shows you have over fitted on training. Maybe regularization can help.
A: This question is old but posting this as it hasn't been pointed out yet:
Possibility 1: You're applying some sort of preprocessing (zero meaning, normalizing, etc.) to either your training set or validation set, but not the other.
Possibility 2: If you built some layers that perform differently during training and inference from scratch, your model might be incorrectly implemented (e.g. are moving mean and moving standard deviation for batch normalization getting updated during training? If using dropout, are weights scaled properly during inference?). This might be the case if your code implements these things from scratch and does not use Tensorflow/Pytorch's builtin functions. 
Possibility 3: Overfitting, as everybody has pointed out. I find the other two options more likely in your specific situation as your validation accuracy is stuck at 50% from epoch 3. Generally, I would be more concerned about overfitting if this was happening in a later stage (unless you have a very specific problem at hand).
A: Adding to the answer by @dk14 . If you are still seeing fluctuations after properly regularising your model, these could be the possible reasons:


*

*Using a random sample from your validation set: It means your validation set at each evaluation step is different, so is your validation-loss. 

*Using a weighted loss-function(which is used in case of highly imbalanced class-problems). At train step, you weigh your loss function based on class-weights, while at dev step you just calculate the un-weighted loss. In such case, though your network is stepping into convergence, you might see lots of fluctuations in validation loss after each train-step. But if you wait for a bigger picture, you can see that your network is actually converging to a minima with fluctuations wearing out.(see the attached images for one such example). 
A: There are few ways to try in your situation. Firstly try to increase the batch size, which helps the mini-batch SGD less wandering wildly. Secondly tuning the learning rate, probably set it smaller. Thirdly, try different optimizer, for instance Adam or RMSProp which are able to adapt learning rates for wrt features. If possible try augmenting  your data. Lastly, try Bayesian neural networks via dropout approximation, a very interesting work of Yarin Gal https://arxiv.org/abs/1506.02158
A: Have you tried a smaller network? Considering your training accuracy can reach >.99, your network seems have enough connections to fully model your data, but you may have extraneous connections that are learning randomly (i.e. overfitting). 
In my experience, I've gotten the holdout validation accuracy to stabilize with a smaller network by trying various networks such as ResNet, VGG, and even simpler networks. 
