I have a dataset that is divided into training and validation dataset. I am using Gaussian Processes to perform hyperparameter optimization. So I am using the accuracy recorded on the validation dataset to tune the hyperparameters of the DNN model. Is that considered cheating? Will the last reported results be considered credible?

Any help is much appreciated!!


As a result of doing that you will also overfit the validation set (the more so the more you tuned the hyperparameters - if you tried two or three configurations, the effect is less than if you did some systematic search e.g. using the Gaussian process approach). The standard solution to this would be to not just have a training and validation set, but a third set (a test set). You would only ever look at the test set once with you very final model after hyperparameter tuning.

  • 3
    $\begingroup$ Depending on computational limitations, it may also be possible to evaluate a hyperparameter configuration via cross validation on the training set. $\endgroup$ Dec 18 '18 at 21:58
  • 1
    $\begingroup$ I've always referred to a non-final test set as a "test set" and validation to be only the final set to test to validate the entire system. Is there a standard on this that I've been ignoring or is it person to person? $\endgroup$
    – Poik
    Dec 19 '18 at 16:20
  • $\begingroup$ @JohnMadden Also depends on data limitations. We have very little to hold out for proper validation for my task, sadly. Cross validation helps in this scenario. $\endgroup$
    – Poik
    Dec 19 '18 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.