Adam is generally regarded as being fairly robust to the choice of hyper parameters, though the learning rate sometimes needs to be changed from the suggested default.
if this is true its a big deal because hyper parameter search can be really important (in my experience at least) in the statistical performance of a deep learning system. Thus, my question is, why is Adam Robust to such important parameters? Specially $\beta_1$ and $\beta_2$?
I've read the Adam paper and it doesn't provide any explanation to why it works with those parameters or why its robust. Do they justify that elsewhere?
Also, as I read the paper, it seems that the number of hyper parameters they tried where very small, for $\beta_1$ only 2 and for $\beta_2$ only 3. How can this be a thorough empirical study if it only works on 2x3 hyper parameters?