# Why do we sample from log space when optimizing learning rate + regularization params?

Since I took Karpathy's CS231n I used the method he mentions on the 5th lecture for hyperparameter optimization of neural networks which samples the learning rate and regularization parameters from the log space randomly.

It seems to work great from experience, but I never understood why that's the right thing to do from the lecture.

I would appreciate an intuitive explanation about why the log space is where we sample from.

Take the learning rate as an example. Let's say we decide to sample uniformly from 0 to 1, then only about 10% of the values would come from 0 to 0.1, and 90% of the values would come from 0.1 to 1. This does not seem to be appropriate because we would definitely like to sample values in the order of $$10^{-2}$$, $$10^{-3}$$, $$10^{-4}$$, $$10^{-5}$$,etc and all of these values fall under the first group that has only 10% chance of being selected.
Instead, if we used a logarithmic scale to sample the values such as from -5 to 0, then values from $$10^{-4}$$, $$10^{-3}$$, $$10^{-2}$$, $$10^{-1}$$, $$10^{0}$$ all have equal chance of being selected.
• Similar to the scenario that I've given previously, if we sample uniformly from from 0 to $10^-5$, then only 10% of the values would come from 0 to $10^-6$ and 90% of the values would come from $10^-6$ to $10^-5$. As such most of the values that are selected would be in the order of $10^-6$, which might not be very helpful. I think the contention here is that it is not outright wrong to search our hyperparameters in linear scale but it is inefficient to do so as compared to using the log scale, especially in the initial phase of our search when we want to sample from a wider range. – Yaofeng Wang Jun 24 '19 at 1:33