1
$\begingroup$

My mathematical statistics theory professor, an expert in the field, said today he doesn't trust statistics enough to use it in practice. He just likes the beauty of the math. He said he would never be able to teach applied statistics in good conscience. Do you trust statistics? If so, why?

The more statistics I learn, the more I realize how much of it is founded on assumptions. We specify assumptions, and then use rigorous mathematics to generate conclusions. Why is this any better than not using advanced statistics? After all, we do not know our assumptions are true. How can we justify hypothesis testing, for example, when we don't know the true distribution? (I ignore descriptive statistics here, which are clearly useful.)

The more I learn, the less confidence I have in statistics and the more faith I have in machine learning, which is not bold enough to make claims like causality.

$\endgroup$

closed as primarily opinion-based by Michael Chernick, Stefan, Martijn Weterings, Scortchi Feb 9 at 6:39

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 5
    $\begingroup$ You think machine learning does not need assumptions? $\endgroup$ – kjetil b halvorsen Feb 9 at 1:06
  • $\begingroup$ Of course it requires assumptions, but there is a clear out-of-sample testing process. Model quality is evaluated based on test error. The same is not commonly done for causal analysis in classical statistics, for example. $\endgroup$ – purpleostrich Feb 9 at 1:08
  • 4
    $\begingroup$ I think that comment is superprovocative which to me just show how opinionated any answer to this question is going to be. $\endgroup$ – Jesper Hybel Feb 9 at 1:45
  • 2
    $\begingroup$ It is absurd to discard a (statistical) model only because it is not entirely correct (no model, also a machine learned pattern, is ever entirely correct). In this way your question asking for convincing fact-focused arguments for the use of statistics is a loaded question (it is wrong to assume/expect complete rigour). You are right however that the mathematical rigour in the methods/models used by statistics can be misleading regarding the degree of rigour in the conclusions that are made based on the use of a certain method/model. $\endgroup$ – Martijn Weterings Feb 9 at 6:57
  • 1
    $\begingroup$ Re causality: recently ML / deep learning moved into the causality field. Also statistics and ML (aka nonparametric statistics) are here to solve real world problems. Understanding / estimating causality is at the core most real world problems that we face. So the fact that statistics tries to answer those questions , rather than just say "we can't do cross validation for it, so we 'll not try to answer these important questions" is good! Besides nonparametric stats / ML is actively working on validation and assumption testing for causal inference $\endgroup$ – Georg M. Goerg Feb 9 at 13:18
2
$\begingroup$

I myself do two types of statistics.

One is statistical analyses of psychological experiments. I agree that the classical NHST paradigm is... problematic. The (almost invarably misunderstood) p value gives the probability of observing a test statistic as extreme as the one actually observed under the null hypothesis, conditional on the model being correct. Now, we are pretty sure that the null hypothesis is not true, otherwise we would not be running the experiment. And the model is almost certainly not correct, given tapering effect sizes in the sense of Burnham & Anderson. So NHST can be said to give an answer to an irrelevant question using an incorrect approach. Yes, I understand how this makes mathematicians queasy.

Then again, it works reasonably well, even in light of the so-called replication crisis. Once you look beyond single studies, a number of things simply fit together too well and make for a coherent whole. So the statistics seem to be doing something right. Also, as a mathematician, one understands the limitations of incorrect models, and the power of low-order approximations. The Taylor rule could be said to hold.

My day job is forecasting retail sales. I mainly use regression type approaches with a dash of Bayes, nothing one would reasonably classify as ML. No p values involved. And here, I trust my statistics because they work. The forecasts are not perfect, but they are again reasonably good. At least good enough to ensure that enough product is on the shelves when the next promotion and price reduction starts.

I studied pure math, and as I said, I understand the misgivings of your professor. Statistics, along with applied math and physics, have an interesting in-between existence. On the one hand, you can derive theoretical results, some of which actually help in the "real world". There is really a bit of art to understanding how far the theory is applicable and when it stops being so, and where to leave pure theory behind in order to achieve results that matter outside it.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.