I'm using random search for hyper-parameter optimization of a machine learning pipeline. For example, for the C and gamma parameter it is recommended to use logarithmically spaced values. Why should I use such values? For example, if I use logarithmic spaced values from $2^{-5}$ to $2^{15}$, then there will be many more values near to $2^{-5}$ (i.e. near zero) than near to $2^{15}$.

  • $\begingroup$ Why not make draws from an extreme value distribution such as Tweedie, Cauchy, and so on (see en.wikipedia.org/wiki/Tweedie_distribution#Examples)? This would almost certainly provide a better fit to heavy-tailed information and their optimization. $\endgroup$ – DJohnson Feb 3 '17 at 15:46

... because logarithmic scale enables us to search a bigger space quickly. In your SVM example, we do not know the range for the hyper-parameter. So, a quicker way is trying dramatically different values, say, 1, 10, 100, 1000, which come from a logarithmic scale.

In addition, I think log scale search is the first step. Suppose that we found C=10 is better than C=1 or C=100; then we can focus on that scale to try a better value.

Another reason is for "regularization" parameters, such as C in svm. It is not too sensitive. In other words, we may not find too much difference with 10 or 15, or 20, but results would be very different from 10 to 1000. That is why we start with log search.

  • $\begingroup$ But does this make sense for random search? $\endgroup$ – machinery Feb 3 '17 at 15:11
  • $\begingroup$ @machinery to do random search, we still need a range. Say from 1 to 1000 uniform ? $\endgroup$ – hxd1011 Feb 3 '17 at 15:16
  • $\begingroup$ Yeah you need a range but should the range be uniform or logarithmic? $\endgroup$ – machinery Feb 3 '17 at 15:48
  • 1
    $\begingroup$ I think the first paragraph is incorrect. Suppose your budget is $k$ different values of $C$. We can search the space $[1,10^3]$ by checking $k$ points equally spaced on a linear scale. If the space is expanded to $[1,10^6]$, this is still true, we can still check $k$ values, except they'll be farther apart. But your point that the difference in performance is more important on a log scale is the heart of it: performance at $C=10\approx11$, but $C=100$ will be different. $\endgroup$ – Sycorax Feb 3 '17 at 15:54
  • 2
    $\begingroup$ We use log scale for hyper-parmaeter optimization because the response function varies on a log scale. Compare a false-color plot of the hyper-parameter surface for RBF length-scale and C on log and linear scales. On a linear scale, in many problems, the plot is basically the same everywhere because the difference between C=10 and C=11 is basically zero, but the difference between C=10 and C=100 is more pronounced. $\endgroup$ – Sycorax Feb 3 '17 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.