Model performance graph is oscillating? I've seen a fair amount of information online regarding the loss graph of a deep learning model oscillating, but not so much about the performance graph. In general, when I "properly" train a neural network, the performance usually increase and then plateaus. However, I've noticed that in some cases the performance itself is very shaky. What is the cause of such behavior and what could be a fix?
One thing that comes to mind is that decreasing the learning rate may help because the variance seems high, but I'm not sure if this is based on any proof. Any tips are appreciated. Thanks.
Edit
Here's an example of what I'm talking about. I would expect that the macro-F1 score should also display similar patterns to micro-F1 despite being lower. However, it's oscillating quite a bit.

 A: With a classification problem one can compute multiple $F_1$ scores for the classes seperately. The micro and macro $F_1$ scores compute a single score for all classes together.
If a few categories have little representation in the validation sample then their individual $F_1$ scores will be noisy and this influences the grouped macro and micro $F_1$ scores. The differences in the weight these two give to different classes makes that the one can be more noisy than the other.

*

*The macro-$F_1$ score computes a non-weighted average of $F_1$ scores for multiple categories.
The noise in the $F_1$ score of one or more small categories influences the macro-f1 score as well.


*The micro-$F_1$ score sums all the false/true negative/positive cases before computing a single $F_1$ score.
The number of false/true negative/positive cases in the categories that are small will be small and so these have relatively less influence on the sum. The micro-$F_1$ score will be relatively less noisy in comparison to the macro-$F_1$ score which gives more weight to the smaller categories.
A: Begin digression:
micro-F1 is just accuracy. A short proof:

*

*accuracy is the number of correct predictions divided by the total number of examples.

*Recall is TP/(TP+FN) that is the accuracy of the positive class (TP+FN are the total number of positive examples).

*Precision is TP/(TP+FP) which is in some way the accuracy when the classifier says "positive" (TP+FP are all examples classified by positive by the classifier).

*F1 is the harmonic mean of precision and recall

*micro recall is $\sum TP_i / (\sum TP_i + \sum FN_i)$ where $i$ are the classes.

*$\sum TP_i $ are all examples correctly classified

*$\sum FN_i$ are all the examples incorrectly classified

*therefore micro-recall is correctly-classified divided by total number of examples, and therefore accuracy

*micro-precision is $\sum TP_i / (\sum TP_i + \sum FP_i)$ .

*$\sum FP_i$ is also all the examples incorrectly classified,

*therefore micro-precision is also the accuracy

*micro-F1 is the harmonic mean of the same value, accuracy, which is also accuracy.

end digression
Now answering the question. @SextusEmpiricus answer show that macro-F1 should be sensitive to small changes in the accuracy of the small classes.

*

*If the set being used to measure the macro-F1 changes , than one will expect larger  changes in the macro-F1. But the problem is that the usual way generating a performance graph is to have a fixed validation set. So I need the OP to confirm that indeed the set being used to generate the measures plotted in the graph is fixed.


*if the set is fixed, then the most likely explanation is that the validationn set is unbalanced  - there are a few classes with large number of examples and a few with few examples. The neural network is is optimizing a global measure, not accuracy but something similar, and in the process it finds that it is worth to be a little more bold with the large classes - if something looks like it is form a large class A, say it is from A. Since A has a lot of examples, this increases the global measure. But this may remove examples that were correctly classified in a small class B. Now B's accuracy (recall) is worse and since there are only a few examples of B, this reduction on its accuracy is large, and there is a large reduction on b's F1. That would explain some reduction on the macro-F1.


*there is no problem with that and is not an indication of error. The NN is optimizing a global measure (like but not exactly accuracy) and in the process these small classes may suffer. But if what you want is a NN with high macro-F1 (and not necessarily high accuracy) you will need to change what the NN is optimizing for.
This is the limit of my knowledge - there has been a recent literature on imbalanced multi-class classification, and I would guess that some of it is about neural networks. Maybe this literature proposes different cost fuctions or different regularizations for the cost function so metics like macro-F1 are "optimized" (the metric is not optimized, the cost function plus the regularization are).
