After >170 deep learning experiments were I did a (almost) full factorial design with >15 factors.

I cannot measure performance with cross validation because that would require to much training of the classifier. Therefore, I measured performance with the holdout method on a validation set (validation set: totaling >1000 examples, 2 classes, ~300 examples in the smaller class).

I now suspect that my highest performance estimates in the validation set are outliers [based on DL knowledge and comparison of performance metrics]. Therefore, I suspect overfitting if I just choose the highest performance estimate in the validation set.

  1. Are there any other approaches for model selection than choosing the highest performance estimate in the model selection experiments (especially if no cross validation is used)?
  2. I thought that instead I could use linear regression to choose the best configuration via the coefficients [performance estimates = dependent variable; independent variable = categorical model choices]. I do suspect that most categorical factors that I study have only small interactions. However, I have never seen this approach in literature before. Are there any problems related to this linear regression approach?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.