After >170 deep learning experiments were I did a (almost) full factorial design with >15 factors.
I cannot measure performance with cross validation because that would require to much training of the classifier. Therefore, I measured performance with the holdout method on a validation set (validation set: totaling >1000 examples, 2 classes, ~300 examples in the smaller class).
I now suspect that my highest performance estimates in the validation set are outliers [based on DL knowledge and comparison of performance metrics]. Therefore, I suspect overfitting if I just choose the highest performance estimate in the validation set.
- Are there any other approaches for model selection than choosing the highest performance estimate in the model selection experiments (especially if no cross validation is used)?
- I thought that instead I could use linear regression to choose the best configuration via the coefficients [performance estimates = dependent variable; independent variable = categorical model choices]. I do suspect that most categorical factors that I study have only small interactions. However, I have never seen this approach in literature before. Are there any problems related to this linear regression approach?