5
$\begingroup$

Most of the recent famous methods coming out of the machine learning, are supervised learning methods like Decision Trees, Random Forests, Deep Learning, SVMs.

The more traditional supervised learning methods, like linear and logistic regression, with or without regularization, have had a long history of analysis of their nuances (eg assumptions for reliable use like normality, confidence intervals, hypothesis tests, optimal estimators).

Though the traditional stats models and the more modern ML ones come out of different disciplines (for statistics theoretically associated with mathematics departments and practically agronomy, medicine, social science, and econometrics, and for machine learning out of computer science with applications in vision, NLP, and AI), they have the same ends.

It seems like the ML models, as wildly successful as they seem, also seem to have very little theoretical support.

In contrast, linear regression can have a p-value analysis of each variable, F-test for the entire fit, has (the classic five assumptions). I've never seen such analysis of the more complicated ML tests.

There doesn't seem to be a treatment of machine learning models with the rigor of analysis of the statistical models. http://www.fharrell.com/post/stat-ml/

Is there any attempt to apply classic statistical analysis techniques to assessing the newer ML regression models?

$\endgroup$

closed as unclear what you're asking by Sycorax, Michael Chernick, usεr11852, Frans Rodenburg, COOLSerdash Apr 22 at 8:18

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ There are four standard regression assumptions. The author mentions no or low multicollinearity which is not an assumption for regression, although people commonly and incorrectly say it is an assumption. I would re evaluate that reference. $\endgroup$ – LSC Apr 21 at 17:12
  • 1
    $\begingroup$ I am not well-versed to answer that but in contrast to Asymptotic theory that usually pertains statistics I would say that ML is attacking a lot of its methods through generalisation bounds. $\endgroup$ – usεr11852 Apr 21 at 17:23
  • $\begingroup$ I think I've added some detail to address the close voters. Surely part of the cause of lack of clarity on my part is lack of knowledge. My motivation for this question is that I feel like success in the complicated methods of ML is offset by lack of statistical rigor (and dually the great rigor in statistics is offset by lack of progress in new more successful methods). Or is it just that historically, methods were devised first and justifications and analysis came way later, and that is just as much the case for Random Forests now as it was for Logistic Regression in the thirties? $\endgroup$ – Mitch Apr 21 at 18:39
  • 2
    $\begingroup$ I've read your edit and comments but it's not clear what properties of machine learning you want proved. It seems you wish to reason by analogy about $p$-values in some manner. But what would that mean for a random forest model? A $p$-value for regression coefs tests the hypothesis that the coefficient is not statistically different from zero. Random forests don't estimate a coefficient for each variable, and it's not clear what hypothesis you are interested in testing. "Machine learning" usually cares more about making good predictions, which is why @usεr11852 mentions generalization bounds. $\endgroup$ – Sycorax Apr 21 at 18:51
  • $\begingroup$ We have this related thread, which might be of interest stats.stackexchange.com/questions/321851/… $\endgroup$ – Sycorax Apr 24 at 16:38
6
$\begingroup$

I guess the main part of an answer depends on what, precisely, you mean by "classical statistical analysis" but if we interpret it broadly to mean applying theorems and results from probability and statistics, then we can come up with a good bibliography.

Three references off the top of my head:


Aside: It's worth remarking that the difference between machine learning and statistics has more to do with marketing rather than any underlying mathematical principles.

For example, random forests were first proposed by Leo Breiman, who was a statistics professor at University of California, Berkeley.

$\endgroup$
  • $\begingroup$ Thank you for those references. I am well aware of the first two. I suppose I wasn't clear about what I don't think has been done for DL, SVM, RF, for example what kind of distribution assumptions are necessary for robust results. $\endgroup$ – Mitch Apr 21 at 17:00
  • 1
    $\begingroup$ Perhaps you could edit your question to clarify what you want to know in specific terms and why the resources you've consulted don't answer your question. $\endgroup$ – Sycorax Apr 21 at 17:02
  • 2
    $\begingroup$ To answer your question -- It's rarely necessary to assume a specific distribution (e.g. normal) for the data. The whole point of these advanced methods is that they're robust to a wide variety of input data types -- so in a sense they are "generic" to a very large class of problems. For example, the chapter in ESL about random forest demonstrates that it has some nice properties without positing a particular distribution for the inputs. Probably the biggest assumption is that the input data is iid, with independence of inputs being the biggest potential problem. $\endgroup$ – Sycorax Apr 21 at 17:12
  • 4
    $\begingroup$ This is a very accurate point that a lot of the “difference” is marketing a hot “new field” or that many of the “ML” methods are statistical methods with the appropriate brakes disabled or ignored. $\endgroup$ – LSC Apr 21 at 17:18

Not the answer you're looking for? Browse other questions tagged or ask your own question.