What is artistic about hyperparameter tuning? I recently started competing at kaggle which exposed me with hyper parameter tuning of models such as GBM,xgboost etc.All the sources that I studied to get started with an approach to tuning invariably started with the line " Hyper parameter tuning is more of an art."
Most of the texts used Gridsearchcv or randomizedsearchcv to brute force through possible values of parameters.There was nothing "artistic" in this approach.
Is there a next level more insightful approach to hyper parameter tuning that I am missing?How to make hyper parameter optimization more insightful?Do all of us just brute force?
 A: Choosing tuning parameters is an optimization problem: we are optimizing some presumably black-box measure of model performance $g$ over some set of points $S$, where each $s \in S$ is a vector of hyperparameters. For example, if we are tuning a polynomial ridge regression we could have $S = \mathbb N \times [0, \infty)$ so for $s \in S$ we have $s  =(d, \lambda)$, and $g(s)$ gives the LOOCV for this particular degree and penalty. Generally our measure of performance involves cross-validation or some other sampling-based procedure, so it is noisy. In all, we have the following:
Find $\hat s = \textrm{argmax}_{s \in S}\  g(s)$ where


*

*$g$ is observed with noise

*we can't make any assumptions on the structure of $g$, i.e. we don't have gradients

*$g$ is possibly very expensive to evaluate

*$g$ is possibly highly non-convex

*$g$ is data-dependent, so its argmax is not necessarily the population's and may be overfit

*$S$ likely has some discrete dimensions, which is very unhelpful as we may not even be able to do approximate gradients by taking a "small step". It may even have some unordered dimensions (e.g. choice of kernel: there isn't an order on the set $\{\textrm{radial}, \textrm{polynomial}\}$)
All of this makes for a very challenging optimization. Popular derivative-free black-box optimizers are Bayesian optimization and particle swarm, both of which have been successfully applied to tuning parameter selection many times.
Now, where does the art come in? Often if you are doing this by yourself on your laptop, and you need your laptop for work tomorrow so you can't just let your particle swarm optimizer run for a week with a bunch of random restarts. You also may not have enough data to optimize extensively, or you'll likely overfit (so you use early stopping as a heuristic regularization). That means you need good starting conditions, so here's where the art comes in. You base your initial conditions on a fine mixture of guessing and prior knowledge (i.e. what worked well last time). Maybe you do a little exploration (I think a lot of people settle on a sort of heuristic Nelder-Mead where they have a simplex or box that they expand and contract) and then after about 10 tries you say "good enough". That's just my experience, but it seems to be consistent with others that I've spoken with who've used similar language to what you're asking about.
