How to measure variance that is due to random noise in training data

Question

I would like to measure the variance of a binary classification model (deep neural network). Say the performance metric of choice is f1-score. There are two sources of variance that I can think about:

1. Variance due to different random seeds used for initialising the model weights (let's call it algorithm variance);
2. Variance due to random noise in the training data (let's call it data variance).

I have got a single training set of annotated examples.

To model algorithm variance I could:

• split the training set into a training and a validation set
• repeat n times:
  • initialise the model with random weights
  • train the model and evaluate it on the validation set to produce the nth f-score, $f_n$.

I can then consider the variance of $f_n$. Does this sound OK?

I am unsure, however, about how to model data variance. What I have in mind:

• Option 1: I am thinking I could do cross-validation. For each fold, I would have a different training and validation set. I could then consider the variance of f-scores across the folds.
• Option 2: Alternatively, I split the initial training set $T$ into a training set $T_s$ and a validation set $V_s$ in advance. Then, I could train the model on n subsets of this smaller training set $T_s$ (i.e. I just throw away some data each time), each time considering the f-score on the fixed validation set $V_s$ that I chose in advance.

So, in Option 1 I have a different validation set each time. In option 2 I have the same one. Intuitively I feel I should have the same one, i.e. that I should go with option 2. But I am not sure, nor can I think of a proper argument.

Thank you.

Answer 1

In neural network training, the sources of variations are plenty. Because the dynamics are like that of a stochastic differential equation, having same data as in option 2 does not guarantee that given the same weights for initialization your model would deterministically converge to same point after 100 epochs or so.

Both option 1 and option 2 are used for model selection, but given that neural network training is expensive, option 2 is more popular. I don't know how large your dataset is, but if a reasonable argument can be made that separating a validation set (say 1–10% of data) does not cause your model to underfit, then I would go with option 2. This is really a concern in learning from very low data (e.g. fitting VGG-16 from scratch using let's say 10% of CIFAR10 data).

Taking option 2, you can try two things:

1. Select different random seed for weight initialization, repeat neural network training under a consistent model selection strategy (e.g. best performance on val set in E epochs, under assumption that E epochs are more than enough to get the model to converge), and measure a response variable (e.g. f1 score as you proposed).
2. Fix a random seed and repeat training, under a similar model selection strategy to measure response variable as in 1) above. This time, you're after variance caused by the dynamics of neural network training. Why is this necessary you ask? Because the gradient flows are different as the order of mini-batches drawn is different. This is a fascinating direction of on-going research. Further, batch-norm will converge to a slightly different mean and sigma for normalizing activations given that order of minibatches will be different, just FYI if your model is using batch-norm. Lastly, don't forget to turn-off dropout.

I would report variance in response variable under both of the above described scenarios.