# How to measure variance that is due to random noise in training data

I would like to measure the variance of a binary classification model (deep neural network). Say the performance metric of choice is f1-score. There are two sources of variance that I can think about:

1. Variance due to different random seeds used for initialising the model weights (let's call it algorithm variance);
2. Variance due to random noise in the training data (let's call it data variance).

I have got a single training set of annotated examples.

To model algorithm variance I could:

• split the training set into a training and a validation set
• repeat n times:
• initialise the model with random weights
• train the model and evaluate it on the validation set to produce the nth f-score, $$f_n$$.

I can then consider the variance of $$f_n$$. Does this sound OK?

I am unsure, however, about how to model data variance. What I have in mind:

• Option 1: I am thinking I could do cross-validation. For each fold, I would have a different training and validation set. I could then consider the variance of f-scores across the folds.
• Option 2: Alternatively, I split the initial training set $$T$$ into a training set $$T_s$$ and a validation set $$V_s$$ in advance. Then, I could train the model on n subsets of this smaller training set $$T_s$$ (i.e. I just throw away some data each time), each time considering the f-score on the fixed validation set $$V_s$$ that I chose in advance.

So, in Option 1 I have a different validation set each time. In option 2 I have the same one. Intuitively I feel I should have the same one, i.e. that I should go with option 2. But I am not sure, nor can I think of a proper argument.

Thank you.