An outsider to ML/DL field; started Udacity Deep Learning course which is based on Tensorflow; doing the assignment 3 problem 4; trying to tune the learning rate with the following config:
- Batch size 128
- Number of steps: enough to fill up 2 epochs
- Sizes of hidden layers: 1024, 305, 75
- Weight initialization: truncated normal with std. deviation of sqrt(2/n) where n is the size of the previous layer
- Dropout keep probability: 0.75
- Regularization: not applied
- Learning Rate algorithm: exponential decay
played around with learning rate parameters; they don't seem to have effect in most cases; code here; results:
Accuracy learning_rate decay_steps decay_rate staircase 93.7 .1 3000 .96 True 94.0 .3 3000 .86 False 94.0 .3 3000 .96 False 94.0 .3 3000 .96 True 94.0 .5 3000 .96 True
- How should I systematically tune learning rate?
- How is learning rate related to the number of steps?