0
$\begingroup$

I have a huge dataset and want to carry out regression, such as gradient boosting. The problem is that the dataset is huge and hyperparameter optimization is computational expensive, especially I use cross validation for that.

Is it OK to do the hyperparameter optimization on subsets of the dataset? Then can I average these hyperparameters from different subsets and use that to train my model on the whole dataset?

Improve this question
$\endgroup$
3
  • $\begingroup$ A more principled option would be to use a global optimization method such as Bayesian Optimization to keep the number of function calls as low as possible. $\endgroup$
    – Sycorax ♦
    Commented Apr 6, 2016 at 1:00
  • 1
    $\begingroup$ How long a single evaluation of your function takes and how long can you spare for estimation? $\endgroup$
    – usεr11852
    Commented Apr 6, 2016 at 1:16
  • $\begingroup$ Related, if not a duplicate: stats.stackexchange.com/questions/233548/… $\endgroup$
    – DJBunk
    Commented Sep 7, 2018 at 16:13

1 Answer 1

Reset to default
3
$\begingroup$

You need to use algorithms which support multi-fidelity evaluations, i.e., which are able to exploit computationally cheap surrogates of your actual/expensive objective function. Instead of running on 100% of your data you will be able to run on say 1% and 10% to get computationally cheap approximates of the optimum of your high-fidelity objective function.

Multi-fidelity approaches can lead to a significant speedup up to several orders of magnitude if low-fidelity evaluations provide some non-random information which might be exploited for high-fidelity evaluations.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.