How to (systematically) tune learning rate having Gradient Descent as the Optimizer? An outsider to ML/DL field; started Udacity Deep Learning course which is based on Tensorflow; doing the assignment 3 problem 4; trying to tune the learning rate with the following config:


*

*Batch size 128

*Number of steps: enough to fill up 2 epochs

*Sizes of hidden layers: 1024, 305, 75

*Weight initialization: truncated normal with std. deviation of sqrt(2/n) where n is the size of the previous layer

*Dropout keep probability: 0.75

*Regularization: not applied

*Learning Rate algorithm: exponential decay


played around with learning rate parameters; they don't seem to have effect in most cases; code here; results:
Accuracy    learning_rate   decay_steps     decay_rate      staircase
93.7        .1              3000            .96             True
94.0        .3              3000            .86             False
94.0        .3              3000            .96             False
94.0        .3              3000            .96             True
94.0        .5              3000            .96             True



*

*How should I systematically tune learning rate?

*How is learning rate related to the number of steps?
 A: Use a gradient descent optimizer. This is a very good overview.
Regarding the code, have a look at this tutorial. This and this are some examples.
Personally, I suggest to use either ADAM or RMSprop. There are still some hyperparameters to set, but there are some "standard" ones that work 99% of the time. For ADAM you can look at its paper and for RMSprop at this slides.
EDIT
Ok, you already use a gradient optimizer. Then you can perform some hyperparameters optimization to select the best learning rate. Recently, an automated approach has been proposed. Also, there is a lot of promising work by Frank Hutter regarding automated hyperparameters tuning.
More in general, have a look at the AutoML Challenge, where you can also find source code by the teams. In this challenge, the goal is to automate machine learning, including hyperparameters tuning.
Finally, this paper by LeCun and this very recent tutorial by DeepMin (check Chapter 8) give some insights that might be useful for your question.
Anyway, keep in mind that (especially for easy problems), it's normal that the learning rate doesn't affect much the learning when using a gradient descent optimizer. Usually, these optimizers are very reliable and work with different parameters.
A: You can automate the tuning of hyper-parameters in a lot of machine learning algorithms themselves, or just the hyperparameters for Gradient Descent optimizer i.e learning rate. 
One library that has been popular for doing this is spearmint.
https://github.com/JasperSnoek/spearmint
A: A very recent automatic learning-rate tuner is given in Online Learning Rate Adaptation with Hypergradient Descent
This method is very straightforward to implement, the core result for SGD is given as:
$\alpha_{t} = \alpha_{t-1} + \beta \nabla f(\theta_{t-1})^T\nabla f(\theta_{t-2}) $
where $\beta$ is a (hyper) hyperparameter. The method also applies to other gradient-based updates ($\textit{e.g.}$ momentum-based methods). No validation set is needed: it only requires storing the previous gradient, $\nabla f(\theta_{t-2})$. The idea is to use the partial derivative of the objective function w.r.t. the learning rate ($\alpha$), to derive an update rule for alpha. 
Anecdotally, I implemented this on top of my existing problem, and observed much better results. I did not tune $\beta$ or $\alpha_0$, but picked from the suggested ranges from the paper.
A: To tune hyperparameters (whether it is learning rate, decay rate, regularization, or anything else), you need to establish a heldout dataset; this dataset is disjoint from your training dataset. After tuning several models with different configurations (where a configuration = a particular choice of each hyperparameter), you choose the configuration by selecting the one that maximizes heldout accuracy.
