In order to define what deep learning is, the learning portion is often listed with backpropagation as a requirement without alternatives in the main stream software libraries and in the literature. There are not many gradient free optimisations are mentioned in deep learning or in general statistical learning. Similarly, in "classical algorithms" (Nonlinear least squares) involves derivatives . In general, gradient free learning in deep learning or classical algorithms are not in the main stream. One promising alternative is simulated annealing [2, 3], so-called 'nature-inspired optimization'.
Is there any inherent theoretical reason that why gradient free deep learning (statistical learning) is not in the main stream? (Or not preferred?)
 Such as Levenberg–Marquardt
 CoolMomentum: a method for stochastic optimization by Langevin dynamics with simulated annealing (2021) Though this is still not fully gradient-free, but does not require auto-differentiation.
Edit 1 Additional references using Ensemble Kalman Filter, showing a derivative free approach:
- Ensemble Kalman Inversion: A Derivative-Free Technique For Machine Learning Tasks arXiv:1808.03620.
- Ensemble Kalman Filter optimizing Deep Neural Networks: An alternative approach to non-performing Gradient Descent springer (manuscript-pdf)
Edit 2 As far as I gather, Yann LeCun does not consider gradient-free learning as part of deep learning ecosystem. "DL is constructing networks of parameterized functional modules & training them from examples using gradient-based optimization." tweet
Edit 3 Ben Bolker's comment on local geometry definitely deserves to be one of the answers.