Is there a general rule for the number of times training data for a model should be resampled to reduce variance?

Here is an example

Run 1:

Sample 1) 0.8431 Sample 2) 0.8430 Sample 3) 0.8431 Sample 4) 0.8432

Standard deviation of samples 0.0000816496580927636

Run 2

Sample 1) 0.55647 Sample 2) 0.4538 Sample 3) 0.65217 Sample 4) 0.6

Standard deviation of samples 0.0841829780893967

In run 1 the results were pretty much the same every time, so I don't see a reason to sample more than once. In run 2 the standard deviation is much higher. Run 2 would require resampling to reduce the model variance and give accurate predictions. How do I decide on the number of times to resample run 2?

  • $\begingroup$ In a run, are you training and running the same ML algorithm on different resamples of the dataset? $\endgroup$
    – gunes
    Jun 29 '21 at 8:23
  • $\begingroup$ assume everything else is the same $\endgroup$
    – Eric
    Jun 29 '21 at 17:54
  • $\begingroup$ Is the model used in Run 1 and Run 2 are the same? Or are they different models? If they're the same, I don't understand what makes Run1 different from Run2 other than randomness. $\endgroup$
    – gunes
    Jun 29 '21 at 20:01
  • $\begingroup$ Run 1 has nothing to do with run 2. In run 1 there was hardly any variance between samples. In run 2 there was a lot more variance between samples. How many times should something like run 2 be resampled to deal with the large amount of variance? $\endgroup$
    – Eric
    Jun 29 '21 at 20:53

This is actually model variance and the results may not change much between iterations. So, if there is inherent variance to the model, it's impossible to decrease it by performing many resamples. You'll only estimate this variance better by increasing the number of resamples.

For example, if your model was outputting a constant, e.g. 5, the variance would be $0$, and increasing the number of resamples wouldn't change this.

Similarly, if your model is totally outputting random numbers with some variance, $\sigma^2$, it's still not possible to decrease it by increasing the number of resamples.

  • $\begingroup$ How do we deal with this model variance? I was under the impression we had to resample many times and take the average. I'm trying to figure out if there is a rule or formula for the number of times resampling should be done. It eats up a ton of memory to resample a dataset 1000 times. I'd like do this as efficiently as possible. $\endgroup$
    – Eric
    Jun 30 '21 at 2:20
  • $\begingroup$ You can't remove model variance by resampling, only discover it. Model variance is overfitting, I'd suggest searching on the forum and web since it's a relatively large problem, and also depends on algorithms. $\endgroup$
    – gunes
    Jun 30 '21 at 8:25
  • $\begingroup$ What should I search for? That's why I am asking here. I don't know what to search for. $\endgroup$
    – Eric
    Jun 30 '21 at 17:38
  • $\begingroup$ It's probably "overfitting" that your model suffers. I'd advise you to ask another question about your model used in Run2, explaining the model in detail, its data, validation/test method, and asking for help. Overfitting problem can't be resolved in general. $\endgroup$
    – gunes
    Jun 30 '21 at 19:22
  • $\begingroup$ so resampling/bootstrapping/bagging whatever you want to call it are not solutions to reduce model variance? $\endgroup$
    – Eric
    Jul 1 '21 at 0:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.