2
$\begingroup$

I have a dataset of 700,000 rows that Im applying random search on. My parameter grid looks like this:

parameters_gbc = {
"loss":["deviance"],
"learning_rate": [0.01, 0.025, 0.075, 0.1, 0.2],
"max_depth":[3,5,8,10],
"max_features":["log2","sqrt"],
"criterion": ["friedman_mse",  "mae"],
"subsample":[0.5, 0.618, 0.8, 0.85, 0.9, 0.95, 1.0],
"n_estimators":[10, 100, 200, 350]
}

I am using 20 iterations for the search:

n_iter_search = 20
random_search_gbc = RandomizedSearchCV(gbc, param_distributions=parameters_gbc,
                                   n_iter=n_iter_search)

I'm not very experienced with this, so how long should I expect the search to be over? How can I make my search more time effective since it has been running for the past 30 hours?

$\endgroup$
0
$\begingroup$

With $20$ iterations you are exploring $\tfrac{20}{1 \times 4 \times 4 \times 2 \times 2 \times 7 \times 4} \approx 1\%$ of the parameter grid, so you would be exploring only a small fraction of the search space. You would need many more iterations then this.

For finding the optimal parameters algorithms smarter then random search (e.g. based on Gaussian processes, or tree-based), should be faster in many cases. Still, you would need many more iterations for reasonable results and since random search has no overhead, then your model is a bottleneck, so this will take some time.

The ultimate solution is to buy, or rent cloud-based) a better computer, with more CPUs, more RAM, and with GPUs (XGBoost and LightGBM have support for GPUs). If this is not an option, you could try training the model on smaller subsets of the data.

$\endgroup$
0
$\begingroup$

Randomized search can take long for a big dataset, this is quite normal. You should check the parameter n_jobs which helps to parallelize the search and make it faster (https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html).

However, in general there are more optimised solutions for this, i.a. bayesian optimization (check out this great blog post with python code:https://thuijskens.github.io/2016/12/29/bayesian-optimisation/). Instead of selecting hyperparameters randomly without any strategy, bayesian optimization tries to find hyperparameters that lead to better results than in the last setting. You approach a better solution step-for-step and probably in less time.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.