I am using neural network algorithms for a relatively large dataset with 1700 obs and 40 features.
I performed optimizing by nested cross validation.
I also wanted to compare 5 algorithm with each other by benchmarking.
When I select about 5 hyperparameters to be tuned (number of nodes, layers, alpha, dropout, epochs),it takes a lot of time for computer to calculate, then I canceled it out.

As Tunning of many of the hyperparameters especially on a large dataset with many features, is so computationally expensive, Is it allowed that we select just a limited subset of hyperparameters (eg.just Num_nodes and dropout) and not all or many of them to be tuned?
I searched it in google and SO questions but did not find the answer.
I appereciate your kind help.

  • 3
    $\begingroup$ You can fix some hyperparameters, sure. A valid response from a boss or a reviewer is, "How did you pick that value?" If you found it through cross validation, that is straightforward to defend. If you just picked because it is your lucky number or your birthday, that might be hard to defend. $\endgroup$
    – Dave
    Apr 19, 2021 at 16:47
  • 2
    $\begingroup$ You don’t need to do grid search, by the way. Randomized search performs very well in less time. There’s also the Bayesian optimization route. $\endgroup$ Apr 19, 2021 at 16:49
  • 1
    $\begingroup$ Something that can be defended is mimicking an architecture that you know works for a similar problem. $\endgroup$
    – Dave
    Apr 19, 2021 at 17:42
  • 1
    $\begingroup$ It's easy to run some simple deep learning, have it complete in a reasonable amount of time, and dismiss claims that deep learning requires beefy hardware. Then you get into doing thousands of cross validation training runs, and that twenty minutes of training time turns into two weeks. $\endgroup$
    – Dave
    Apr 19, 2021 at 17:56
  • 2
    $\begingroup$ No, I'm saying you can randomly search to tune over all of your desired parameters, but when you're time-constrained, you're limited in by how many parameter tuples that you can test in the allotted time. $\endgroup$
    – Sycorax
    Apr 19, 2021 at 21:21


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.