2
$\begingroup$

Since I took Karpathy's CS231n I used the method he mentions on the 5th lecture for hyperparameter optimization of neural networks which samples the learning rate and regularization parameters from the log space randomly.

It seems to work great from experience, but I never understood why that's the right thing to do from the lecture.

I would appreciate an intuitive explanation about why the log space is where we sample from.

$\endgroup$
1
$\begingroup$

Hyperparameters such as learning rate and regularization term tend to be very small positive numbers. When we sample them, we would like to sample values from all the orders of magnitude in a given interval.

Take the learning rate as an example. Let's say we decide to sample uniformly from 0 to 1, then only about 10% of the values would come from 0 to 0.1, and 90% of the values would come from 0.1 to 1. This does not seem to be appropriate because we would definitely like to sample values in the order of $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$,etc and all of these values fall under the first group that has only 10% chance of being selected.

Instead, if we used a logarithmic scale to sample the values such as from -5 to 0, then values from $10^{-4}$, $10^{-3}$, $10^{-2}$, $10^{-1}$, $10^{0}$ all have equal chance of being selected.

If what I've said does not make much sense, I would recommend watching https://www.youtube.com/watch?v=cSoK_6Rkbfg&list=PLkDaE6sCZn6Hn0vK8co82zjQtt3T2Nkqc&index=25 .

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ For the sake of the discussion, than why not sample uniformly from 0 to 0.00001? $\endgroup$ – bluesummers Jun 23 '19 at 6:17
  • $\begingroup$ Similar to the scenario that I've given previously, if we sample uniformly from from 0 to $10^-5$, then only 10% of the values would come from 0 to $10^-6$ and 90% of the values would come from $10^-6$ to $10^-5$. As such most of the values that are selected would be in the order of $10^-6$, which might not be very helpful. I think the contention here is that it is not outright wrong to search our hyperparameters in linear scale but it is inefficient to do so as compared to using the log scale, especially in the initial phase of our search when we want to sample from a wider range. $\endgroup$ – Yaofeng Wang Jun 24 '19 at 1:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.