Why is it almost impossible to optimize both prediction and explanation? 
In my view, it is almost impossible to successfully optimize both accurate prediction of previously unseen observations, and understanding of the phenomenon. Better to focus appropriately.---ljubomir

ljumboir wrote this comment to explain an answer he wrote that compares Machine Learning to Statistics:

I'll argue that the distinction between machine learning and statistical inference is clear. In short, machine learning = prediction of future observations; statistics = explanation.

Ijumboir's answer is excellent, and his distinction between prediction and explanation makes a lot of intuitive sense. But I am confused why that distinction even exist in the first place. Why is it so hard to both predict the future and explain the present? I would have thought that if you can predict something, you can also explain it (and vice versa), but it seems that isn't true in the real world...and I'm curious why that's the case.
 A: I don't agree with the characterization of the distinction between machine learning and statistics.  Both machine learning and statistics are about both prediction and explanation and have tools for both.  Arguably machine learning is a branch of computational statistics, so the distinction between the two isn't straightforward either.
As to "Why is it so hard to both predict the future and explain the present?", in my opinion it is mostly because we are only capable of understanding (relatively) simple explanations (e.g. smallish decision trees, linear models) which may not be sufficiently complex to represent the true underlying behaviour of the system being modelled.  "All models are wrong, but some are useful" - one way in which they can be useful is by being simple enough that we can understand them, another is being complicated enough that they give accurate predictions.  Horses for courses.
Obviously there are some examples where the optimal explanatory model and the optimal predictive model coincide, e.g. classification problems where theory suggests the class conditional densities are Gaussian, a linear model will be both explanatory and likely to give good predictive performance (as it is the Bayes optimal model).  However in most circumstances this won't be the case as the problem is less well constrained and model assumptions unlikely to be completely valid.  "Everything should be as simple as possible, but no simpler", attributed to Albert Einstein comes to mind!
BTW when seeking an explanatory model, it is often a good idea to fit a complex "machine-learning" black-box model as well, if only to get an indication of the amount the explanatory model is not explaining.
A: I don't agree. If you understand something, you are also able to predict it's behavior. I guess that what ljubomir meant is that if you are interested in building a model, to describe and understand some phenomenon, you want to keep it sparse and simple, because complicated models do not help you to better understand complicated reality (see Occam's razor). On another hand, if you only want to make predictions, you do not care if your model is complicated and not easily interpretable if only it predicts correctly. This however doesn't mean that complicated models are better for making predictions! Complicated model can overfit the data, while simple model may be more robust and so better serve for long-term predictions. To give one more example, in some areas you can build very accurate models of phenomenons of interest, that can give perfect predictions (e.g. a mechanistic model of some physical process).
