How to approach loss function based on simulation results in modern ML framework I am trying to figure out the best way to train a NN with using a particle simulations results as the loss function.
I may be simply searching the wrong terms, but it seems like most frameworks require all operations to be built into a graph/tape for differentiation. This does not seem compatible with using a simulation with semi-randomized effects occurring.
How would one approach a problem like this?
 A: In order to train a neural network with backpropagation / gradient descent, it's generally necessary for the loss to be differentiable with respect to the  parameters of the neural network.
If you have a differentiable physics simulator, you can have a loss which depends on the output of the simulation. However, if your physics engine is not differentiable in this way, you would have to rely on some alternative techniques:

*

*You could try reparameterizing all randomized effects. For example, if your simulation has $z \leftarrow \mathcal{N}(x,1)$, you can rewrite this as $z \leftarrow x + \mathcal{N}(1)$, such that $z$ is still differentiable wrt $x$. Similar reparameterizations exist for bernoulli and categorical distributions as well (although those are harder to work with).


*You could use the "score function estimator": $\nabla_\theta E_x [f(x)] = E_x[f(x) \nabla_\theta \log p(x;\theta)]$ where $\theta$ are the parameters of your neural net, $p(x; \theta)$ is the density your model assigns to $x$, and $f(x)$ is a possibly stochastic loss that your simulation assigns to $x$.


*Finally you might also try a number of related tricks for backpropagating through non-differentiable models such as VIMCO, REBAR, and RELAX.
