I have come across several articles online stating that mathematicians don't really understand how machine learning works. One example I can come up with right now is this. There are several others too but can't remember enough to mention. Some other article mentioned that they are like black boxes producing elegant solutions without providing any clear insight as to how the solution was arrived at. Could someone explain it in a bit more detail?
E. g. neural networks function exactly like black boxes. We know general grounds on which they work and how to train them. But we don't know what features does neural network compute on hidden layers. Well, we can guess and test our guesses. But there is no guarantee that we well succeed in our guessing or that discovered features will describe some known characteristics of the phenomenon we are using neural network for.
It is near to impossible to convert trained neural network to a bunch of if-conditions. And if you perform such conversion, then god help you with figuring out meaningful names for features neural network learned in it's hidden layers.