Are line search methods used in deep learning? Why not? A lot of tutorials online talk about gradient descent and almost all of them use a fixed step size (learning rate $\alpha$). Why is there no use of line search (such as backtracking line search or exact line search)?
 A: This question was asked in early 2018, but if you still wait for an answer then: Yes, now there are some implementation of line search in DNN with good performances. See:
https://arxiv.org/abs/1808.05160 (published in 2 journals)
and more recently, by a different group:
https://arxiv.org/abs/1905.09997 (published in ICML).
It is important to note that instead of many claims: 1st, the theoretical results in the stochastic settings are not strong enough (for convex functions only, most DNN are highly nonconvex) and 2nd, implementation in DNN requires some modifications to suit with mini-batch practice (stochastic optimisation, while close, is not the same as mini-batch practice).
P.S. The performances were done on CIFAR10 and CIFAR100. Now more resources and personnel needed to check with a larger range of datasets/DNN.
You can also look at the Wikipedia page:
https://en.wikipedia.org/wiki/Backtracking_line_search
A: Vanilla gradient descent can be made more reliable using line searches; I've written algorithms that do this and it makes for a very stable algorithm (although not necessarily fast). 
However, it makes almost no sense to do a line search for stochastic gradient methods. The reason I say this is that if we do a line search based on minimizing the full loss function, we've immediately lost one of the main motivations for doing stochastic methods; we now need to compute the full loss function for each update, which typically has computational cost comparable to computing the full first derivative. Given that we wanted to avoid computing the full gradient because of computational costs, it seems very unlikely that we want be okay with computing the full loss function. 
Alternatively, you might think of doing something like a line search based on your randomly sampled data point. However, this isn't a good idea either; this will tell you nothing about whether you have stepped too far (which is the main benefit of line searches). For example, suppose you are performing logistic regression. Then each outcome is simply a 0 or 1, and for any single sample, we trivially get perfect separation so the optimal solution for our regression parameters based on the sample of 1 is trivially $-\infty$ or $\infty$ by the Hauck Donner effect. That's not good. 
EDIT
@DeltaIV points out that this also applies to mini-batch, not just individual samples.
A: The tutorials talk about gradient descent presumably because it is one of the simplest algorithms used for optimization, so it is easy to explain. Since most of such tutorials are rather brief, they focus on simple stuff. There are at least several popular optimization algorithms beyond simple gradient descent that are used for deep learning. Actually people often use different algorithms then gradient descent since they usually converge faster. Some of them have non-constant learning rate (e.g. decreasing over time). For review of such algorithms you can check the An overview of gradient descent optimization algorithms post by Sebastian Ruder (or the arXived paper).
