2
$\begingroup$

I am implementing a R function that applies thousands of different models configurations (different configurations for Neural Networks, SVM, RandomForest, k-NN etc) to try to come up with the most accurate model among all.

The point is: as I am testing thousands of different models, it can happen that a model result in good accuracy, just by chance. For example, if you test thousands of random binary classification models using a relatively small validation set (which is my case), you will end up finding some good accuracy, just by chance. This effect is illustrated in the following R code, in which the max accuracy reaches 83% :

  set.seed(1)

  # No of models being tested
  nModels = 10000

  # No of classes of classification: binary
  nClass = 2

  # Length of validation set
  valSetLen = 30

  # Vector with classes, from which the Classifications will be sampled
  classVec = c(1:nClass)

  accur=matrix()

  # simulate the dependent variables of the validation set
  trueClas = sample(classVec,valSetLen, replace=TRUE)

  for (ii in c(1:nModels)){

     # Random Model: randomize between the two classes
     predClas = sample(classVec,valSetLen, replace=TRUE)

     # Accuracy of random model
     accur[ii] = length(which(trueClas==predClas)) / valSetLen
  }

  hist(accur)
  print(max(accur))

Obs.: the random model is just an example!

In my real application, I am using bootstrap estimation of mean and variance of accuracy, calculated using a k-fold cross validation, but I am not sure if this is enough. If the number of models tested is high enough, it seems that maybe I will choose a model that was somehow benefited from randomness.

This problem seems analogous to multiple hypothesis testing in statistics, in which you take into account the fact that many hypothesis are being tested.

Any ideas on how to avoid this problem?

$\endgroup$

1 Answer 1

1
$\begingroup$

Selecting best model from some pool, regardless how, is a meta-model, which you can test using an independent test set or cross-validation. In case this pool is large enough to cause overfitting, you will simply see poor accuracy of the meta-model.

$\endgroup$
1
  • $\begingroup$ Good point @mbq, an independent test set can help to reduce this effect. Still, I wonder if there is any formal test that would give me the following: even though my best model found is 83%, given the size of validation/ test set, the number of classes and the number of models tested, there is 95% probability of this model to have accuracy between X and Y range. $\endgroup$ Jan 6, 2016 at 9:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.