What is artistic about hyperparameter tuning?

I recently started competing at kaggle which exposed me with hyper parameter tuning of models such as GBM,xgboost etc.All the sources that I studied to get started with an approach to tuning invariably started with the line " Hyper parameter tuning is more of an art." Most of the texts used Gridsearchcv or randomizedsearchcv to brute force through possible values of parameters.There was nothing "artistic" in this approach.

Is there a next level more insightful approach to hyper parameter tuning that I am missing?How to make hyper parameter optimization more insightful?Do all of us just brute force?

• When people say "is more of an art", they usually just mean that it isn't easy to do mechanically. Sometimes it's just a nice way to say "you have to **** around with it by hand until something works". – Stumpy Joe Pete Jun 27 '17 at 0:12
• A search takes the art out of parameter selection; it doesn't take the art out of choosing the search space itself. – user0 Jun 27 '17 at 1:32

Choosing tuning parameters is an optimization problem: we are optimizing some presumably black-box measure of model performance $g$ over some set of points $S$, where each $s \in S$ is a vector of hyperparameters. For example, if we are tuning a polynomial ridge regression we could have $S = \mathbb N \times [0, \infty)$ so for $s \in S$ we have $s =(d, \lambda)$, and $g(s)$ gives the LOOCV for this particular degree and penalty. Generally our measure of performance involves cross-validation or some other sampling-based procedure, so it is noisy. In all, we have the following:

Find $\hat s = \textrm{argmax}_{s \in S}\ g(s)$ where

1. $g$ is observed with noise

2. we can't make any assumptions on the structure of $g$, i.e. we don't have gradients

3. $g$ is possibly very expensive to evaluate

4. $g$ is possibly highly non-convex

5. $g$ is data-dependent, so its argmax is not necessarily the population's and may be overfit

6. $S$ likely has some discrete dimensions, which is very unhelpful as we may not even be able to do approximate gradients by taking a "small step". It may even have some unordered dimensions (e.g. choice of kernel: there isn't an order on the set $\{\textrm{radial}, \textrm{polynomial}\}$)

All of this makes for a very challenging optimization. Popular derivative-free black-box optimizers are Bayesian optimization and particle swarm, both of which have been successfully applied to tuning parameter selection many times.

Now, where does the art come in? Often if you are doing this by yourself on your laptop, and you need your laptop for work tomorrow so you can't just let your particle swarm optimizer run for a week with a bunch of random restarts. You also may not have enough data to optimize extensively, or you'll likely overfit (so you use early stopping as a heuristic regularization). That means you need good starting conditions, so here's where the art comes in. You base your initial conditions on a fine mixture of guessing and prior knowledge (i.e. what worked well last time). Maybe you do a little exploration (I think a lot of people settle on a sort of heuristic Nelder-Mead where they have a simplex or box that they expand and contract) and then after about 10 tries you say "good enough". That's just my experience, but it seems to be consistent with others that I've spoken with who've used similar language to what you're asking about.

• Thanks for the reply!Can you suggest how can I apply Nelder-Mead method on regression models like GradientBoostingRegressor in sklearn?Sure there is a scipy implementation but it would not work with the cost function object of GradientBoostingRegressor.(Due to presence of continuous as well as integral hyperparametrs and also cost function existing as an object) – Jasdeep Singh Chhabra Jun 27 '17 at 5:35
• Yeah, combinations of discrete and continuous variables is an issue for a lot of default optimizers although of course there are plenty of algorithms out there that can handle that. What I meant by "heuristic Nelder-Mead" is that you're not actually running an official algorithm, you're just trying points manually but in a manner reminiscent of a Nelder-Mead-style search. – jld Jun 27 '17 at 14:37
• can you please have a look at stats.stackexchange.com/questions/287394/… – Jasdeep Singh Chhabra Jun 28 '17 at 8:35