How to make a Neural Network(NN) learn when it is an input to an non-differentiable function? I have reinforcement/Imitation learning problem where I have to learn the Policy/controller Network ${\pi}$ for the system to be controlled.
The final loss function I got involves a Black box function F(only I/O and so non-differentiable). Which is
$$  L_{\pi} = || Y_{true} - F(X_{true},    NN_{\pi}(X_{true}) ) ||^2 $$
Where $Y_{true}$ and $X_{true}$ are training data(inputs). $NN_{pi}$(Policy network) is the neural network which needs to be learned. Basically I have to minimize the loss by leaning learn NN_pi such that $F(X_{true}$,  $NN_{\pi}(X_{true}))$ equals  $Y_{true}$.
Now the issue is we cannot backpropagate as the we cannot find gradient of L (as F is its an oracle function we cannot differentiate).
So I tried using Finite difference method to find the gradient of $L_{\pi}$. Ie.
\begin{equation}
    \nabla_\pi F(x,NN_\pi(x)):=\\ \left( \frac{F(x, NN_\pi(x)+\delta) - F(x, NN_\pi(x)-\delta)}{2\delta} \right)\nabla_\pi NN_\pi(x)
\end{equation}
And also the controller/policy has some constraints that its o/p should be in$(-\alpha , \alpha)$ range. So I have bounded the final o/p of $NN_\pi$ with tanh activation function.
But the learning is not happening after 2 or 3 steps.
So is it not possible to learn $NN_\pi(x)$ this way?. or Can it be possible to learn $NN_{pi}$without finding the gradient of F, with any other methods?
 A: Answering the general question, using back-propagation and gradient-based optimization algorithms is not the only possible way. There is a whole family of derivative-free optimization algorithms and applying them to neural networks is an open area of research (if you search for it, you'd find many papers). Derivative-free optimization algorithms don't need the functions to be differentiable. Why then are they not commonly used? The major reason is that they are often slower in convergence time as compared to algorithms that use derivatives. In fact, neural networks are designed to be differentiable, so that we can efficiently train them. They also do not necessarily scale that well, are often not implemented out-of-the-box, and there is no hardware optimized for using them as with GPUs.
If you look at the examples, derivative-free optimization can work for neural networks. Aly et al (2019) found out that

LS outperformed
SGD in terms of final accuracy and loss metrics. However,
SGD was found to be up to 50X faster than LS.

Finally, there is nothing special or magical in gradient descent that makes it "the best" algorithm for the problem, it is just popular because it proved to work well for many problems. While it's not an example of a derivative-free approach, Ali Rahimi (2017) in his Test-of-Time Award NeurIPS talk gave an example where gradient descent (and ADAM) fail miserably as compared to using Levenberg–Marquardt algorithm. This shows that other approaches are possible, though less studied.
