I am implementing a R function that applies thousands of different models configurations (different configurations for Neural Networks, SVM, RandomForest, k-NN etc) to try to come up with the most accurate model among all.

The point is: as I am testing thousands of different models, it can happen that a model result in good accuracy, just by chance. For example, if you test thousands of random binary classification models using a relatively small validation set (which is my case), you will end up finding some good accuracy, just by chance. This effect is illustrated in the following R code, in which the max accuracy reaches 83% :

  set.seed(1)

  # No of models being tested
  nModels = 10000

  # No of classes of classification: binary
  nClass = 2

  # Length of validation set
  valSetLen = 30

  # Vector with classes, from which the Classifications will be sampled
  classVec = c(1:nClass)

  accur=matrix()

  # simulate the dependent variables of the validation set
  trueClas = sample(classVec,valSetLen, replace=TRUE)

  for (ii in c(1:nModels)){

     # Random Model: randomize between the two classes
     predClas = sample(classVec,valSetLen, replace=TRUE)

     # Accuracy of random model
     accur[ii] = length(which(trueClas==predClas)) / valSetLen
  }

  hist(accur)
  print(max(accur))

In my real application, I am using bootstrap estimation of mean and variance of accuracy, but I am not sure if this is enough. Any ideas on how to avoid this problem?

I am implementing a R function that applies thousands of different models configurations (different configurations for Neural Networks, SVM, RandomForest, k-NN etc) to try to come up with the most accurate model among all.

The point is: as I am testing thousands of different models, it can happen that a model result in good accuracy, just by chance. For example, if you test thousands of random binary classification models using a relatively small validation set (which is my case), you will end up finding some good accuracy, just by chance. This effect is illustrated in the following R code, in which the max accuracy reaches 83% :

  set.seed(1)

  # No of models being tested
  nModels = 10000

  # No of classes of classification: binary
  nClass = 2

  # Length of validation set
  valSetLen = 30

  # Vector with classes, from which the Classifications will be sampled
  classVec = c(1:nClass)

  accur=matrix()

  # simulate the dependent variables of the validation set
  trueClas = sample(classVec,valSetLen, replace=TRUE)

  for (ii in c(1:nModels)){

     # Random Model: randomize between the two classes
     predClas = sample(classVec,valSetLen, replace=TRUE)

     # Accuracy of random model
     accur[ii] = length(which(trueClas==predClas)) / valSetLen
  }

  hist(accur)
  print(max(accur))

Obs.: the random model is just an example!

In my real application, I am using bootstrap estimation of mean and variance of accuracy, calculated using a k-fold cross validation, but I am not sure if this is enough. If the number of models tested is high enough, it seems that maybe I will choose a model that was somehow benefited from randomness.

This problem seems analogous to multiple hypothesis testing in statistics, in which you take into account the fact that many hypothesis are being tested.

Any ideas on how to avoid this problem?