4
$\begingroup$

In my view, it is almost impossible to successfully optimize both accurate prediction of previously unseen observations, and understanding of the phenomenon. Better to focus appropriately.---ljubomir

ljumboir wrote this comment to explain an answer he wrote that compares Machine Learning to Statistics:

I'll argue that the distinction between machine learning and statistical inference is clear. In short, machine learning = prediction of future observations; statistics = explanation.

Ijumboir's answer is excellent, and his distinction between prediction and explanation makes a lot of intuitive sense. But I am confused why that distinction even exist in the first place. Why is it so hard to both predict the future and explain the present? I would have thought that if you can predict something, you can also explain it (and vice versa), but it seems that isn't true in the real world...and I'm curious why that's the case.

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ Is it? So would you agree that understanding something makes it actually harder to predict it's behaviour..? I wouldn't... $\endgroup$
    – Tim ♦
    Mar 24, 2017 at 16:56
  • 4
    $\begingroup$ Ljubomirs answer may be excellent for you, but it is far from the top answers from the perspective of the Cross Validated community. The distinction is far from being as clear as it is portrayed. $\endgroup$
    – Richard Hardy
    Mar 24, 2017 at 16:56
  • $\begingroup$ To be fair, there is a philosophy of science literature dedicated to this precise distinction. For example: Aligica, P. D. (2003). Prediction, explanation and the epistemology of future studies. Futures, 35(10), 1027–1040. Hanson, N. R. (1959). On the Symmetry Between Explanation and Prediction. The Philosophical Review, 68(3), 349–358. Rescher, N. (1958). On Prediction and Explanation. The British Journal for the Philosophy of Science, 8(32), 281–290. Scheffler, I. (1957). Explanation, Prediction, and Abstraction. The British Journal for the Philosophy of Science, 7(28), 293–309. $\endgroup$
    – Alexis
    Jul 28, 2019 at 15:42

2 Answers 2

Reset to default
8
$\begingroup$

I don't agree with the characterization of the distinction between machine learning and statistics. Both machine learning and statistics are about both prediction and explanation and have tools for both. Arguably machine learning is a branch of computational statistics, so the distinction between the two isn't straightforward either.

As to "Why is it so hard to both predict the future and explain the present?", in my opinion it is mostly because we are only capable of understanding (relatively) simple explanations (e.g. smallish decision trees, linear models) which may not be sufficiently complex to represent the true underlying behaviour of the system being modelled. "All models are wrong, but some are useful" - one way in which they can be useful is by being simple enough that we can understand them, another is being complicated enough that they give accurate predictions. Horses for courses.

Obviously there are some examples where the optimal explanatory model and the optimal predictive model coincide, e.g. classification problems where theory suggests the class conditional densities are Gaussian, a linear model will be both explanatory and likely to give good predictive performance (as it is the Bayes optimal model). However in most circumstances this won't be the case as the problem is less well constrained and model assumptions unlikely to be completely valid. "Everything should be as simple as possible, but no simpler", attributed to Albert Einstein comes to mind!

BTW when seeking an explanatory model, it is often a good idea to fit a complex "machine-learning" black-box model as well, if only to get an indication of the amount the explanatory model is not explaining.

Improve this answer
$\endgroup$
2
  • 5
    $\begingroup$ It's my opinion that this answer doesn't disagree with ljumboir's statement. It's true you can get predictions from a simple, understandable model. But his point was about optimal prediction. For problems with lots of data (both rows and columns), it seems very unlikely that the optimal predictive model will be understandable for the reasons you state. That said, I don't endorse ljubomir's main premise that "prediction = ML, inference = statistics" $\endgroup$
    – Cliff AB
    Mar 25, 2017 at 19:57
  • $\begingroup$ Thanks @CliffAB, it was only the last bit I was disagreeing with, I'll edit my answer to reflect that. $\endgroup$
    – Dikran Marsupial
    Mar 27, 2017 at 8:34
4
$\begingroup$

I don't agree. If you understand something, you are also able to predict it's behavior. I guess that what ljubomir meant is that if you are interested in building a model, to describe and understand some phenomenon, you want to keep it sparse and simple, because complicated models do not help you to better understand complicated reality (see Occam's razor). On another hand, if you only want to make predictions, you do not care if your model is complicated and not easily interpretable if only it predicts correctly. This however doesn't mean that complicated models are better for making predictions! Complicated model can overfit the data, while simple model may be more robust and so better serve for long-term predictions. To give one more example, in some areas you can build very accurate models of phenomenons of interest, that can give perfect predictions (e.g. a mechanistic model of some physical process).

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ To emphasize further: if you only care about predictions, you're willing to use a complex, uninterpretable model. That doesn't mean it's best to be complicated: a very complex model is likely to overfit. But neither does it mean that the best predictive model is always a simple interpretable one: you need to trade off between the bias of a simple model and the variance of a complex one in whatever way works best for prediction on that problem, independently of how well a human can grasp what the model's doing. $\endgroup$
    – Danica
    Mar 25, 2017 at 23:32
  • 2
    $\begingroup$ Of course, if a human can really understand what's going on in a problem, they may be better-positioned to handle changes if the problem shifts over time. But that doesn't at all mean that a simple interpretable model is, by itself, more robust to problem drift over time. For example, a more complex model might be able to use many redundant features to get a robust encoding of the problem where a model targeted at interpretability might depend on only a few, and thus be more vulnerable to shifts in the dependence on any single feature. $\endgroup$
    – Danica
    Mar 25, 2017 at 23:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.