Why do we sample from log space when optimizing learning rate + regularization params? Since I took Karpathy's CS231n I used the method he mentions on the 5th lecture for hyperparameter optimization of neural networks which samples the learning rate and regularization parameters from the log space randomly.
It seems to work great from experience, but I never understood why that's the right thing to do from the lecture.
I would appreciate an intuitive explanation about why the log space is where we sample from.
 A: Hyperparameters such as learning rate and regularization term tend to be very small positive numbers. When we sample them, we would like to sample values from all the orders of magnitude in a given interval. 
Take the learning rate as an example. Let's say we decide to sample uniformly from 0 to 1, then only about 10% of the values would come from 0 to 0.1, and 90% of the values would come from 0.1 to 1. This does not seem to be appropriate because we would definitely like to sample values in the order of $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$,etc and all of these values fall under the first group that has only 10% chance of being selected. 
Instead, if we used a logarithmic scale to sample the values such as from -5 to 0, then values from $10^{-4}$, $10^{-3}$, $10^{-2}$, $10^{-1}$, $10^{0}$ all have equal chance of being selected.
If what I've said does not make much sense, I would recommend watching https://www.youtube.com/watch?v=cSoK_6Rkbfg&list=PLkDaE6sCZn6Hn0vK8co82zjQtt3T2Nkqc&index=25 .
