5
$\begingroup$

Are there any approaches to checking diagnostics of statistical models, in particular linear regression, by machine learning methods? Many of the standard frequentist tests for numerical estimation of residual diagnostics are insufficient in my opinion. Therefore, I am looking for advanced methods to assess assumptions.

$\endgroup$
8
  • $\begingroup$ So before running linear regression you would use machine learning algorithm to test the assumptions of regression. (1) What than would you use to test the assumptions of the ML model..? (2) ML is not designed for testing anything but for making predictions. $\endgroup$
    – Tim
    Feb 5, 2018 at 12:07
  • $\begingroup$ I'd say that the canonical machine learning approach to such problems would be to fit linear regression model and check how well does it make predictions. $\endgroup$
    – Tim
    Feb 5, 2018 at 12:09
  • $\begingroup$ I do not quite know how statistical models and "machine learning" models are separate. You also indirectly imply a Bayesian approach, e.g. BIC by saying "Many of the standard frequentist tests for numerical estimation of residual diagnostics are insufficient" so your question, as it stands, is not very clear to me. $\endgroup$
    – Zhubarb
    Feb 5, 2018 at 12:11
  • $\begingroup$ @Tim (1) I would first fit the regression model and then check assumptions and influencing factors (res. indp. + normality, var inflation...). The easiest way to look at these are in my opinion graphical assessments (Q-Q etc). I browsed standard several textbooks for good numerical tests. However, I do not like them. I am hoping for modern methods to overcome their disadvantages (2) I agree that ML is primarily used for making predictions. I was hoping someone came up with some clever idea to reuse the methods in another context. $\endgroup$ Feb 5, 2018 at 12:24
  • $\begingroup$ @Berkmeister You are right. I was not clear what I mean. with "statistical models" I mean the old-school standard methods (LM, ANOVA, GLM) and with with "machine learning" I mean anything associated with more modern state-of-the-art classification, regression methods. I does not need to be, though (ML was the first that popped into my mind). I read an article on Bayesian Network methods for assessment of independence, that I would say are unconventional in this context but vaguely approach the problem I am interested in. $\endgroup$ Feb 5, 2018 at 12:35

1 Answer 1

3
$\begingroup$

If you wanted to test somehow the assumptions of linear regression using some method that employs machine learning, you would first need to fit the machine learning algorithm to your data. You would face the problem of assessing the fit of the machine learning algorithm to the data and you would possibly need to check the assumptions of the machine learning algorithm that was used. In machine learning we often do not check the assumptions, people often do not even state them explicitly, but it is not true that those methods do not have any assumptions -- any method does. So by employing machine learning in here, you change the problem definition from checking the assumptions of method A, to checking the assumptions of method B, to verify is assumptions of method A are met. So now two things may go wrong: you may wrongly assume that method B has "converged", or you may make wrong conclusions from the output of method B. Now instead of single test that went wrong, two things may fail! That was the first problem.

The second problem is that when fitting machine learning algorithms, you need to choose, and/or tune the hyperparameters of the model, prepare the features etc., so the result depends on your actions. You don't want to have a "test" that depends on your actions (i.e. if you believe the hypothesis is true, you tune the parameters until the test proves your hypothesis and if you don't, you stop with using default parameters and proclaim you win).

The third problem is that machine learning algorithms are not designed for hypothesis testing. They are designed for classifying, making predictions, clustering etc. They will make their predictions "at all cost", leading to problems like overfitting if something goes wrong. They are not designed for making optimal decisions, since they do not optimize anything that is related to making such decisions (unless you made a classification problem of it, but I'd still argue that it is not how you do testing). Hypothesis tests are designed for testing. Machine learning algorithm return predictions and to make any decision based on the predictions, you need to interpret them. Tests give you clear-cur criteria for this, machine learning don't, so you'd rely on more or less subjective interpretations of the results.

$\endgroup$
6
  • $\begingroup$ Thanks for the reply. I fully agree with all of your points! in r.g.t. paragraph 1: Having two models makes diagnoses more complex. However, it seems that ML methods are often solely judged by their success rate. Once such a model is trained it should not be too much of a hustle?! In regard, to your second paragraph I also agree! That is a big problem! In regard to the third paragraph: I share your view that ML is designed for prediction. However, couldn't one rephrase the problem in context of prediction: .... $\endgroup$ Feb 10, 2018 at 15:53
  • $\begingroup$ ... Given the training sets should we predict the residuals of the new data set as member of the family "assumptions met" or "ass. not met"? Further, there are tools to overcome problems such as overfitting. Finally, your point on subjectivity seems analogous to the frequentist criticism of priors in inference. My response is analogous to that of Bayesians: I think obvious subjectivity is better than covered subjectivity. After all, hypothesis testing and graphical analysis of assumptions are not (entirely) objective either. $\endgroup$ Feb 10, 2018 at 15:59
  • $\begingroup$ Overall, I am fully aware that my questions asks for unorthodox techniques (e.g. using Bayesian tools in frequentist regression inference). I cannot judge if such approach is possible, at all. My question was more directed to seeing whether there have been developments in this area that haven't made their way into the textbooks yet. After reading your reply it seems as if they haven't :) Thanks for your effort to elaborate! $\endgroup$ Feb 10, 2018 at 16:05
  • $\begingroup$ @CarolEisen but when exactly is your model "trained"? What if you fit two different models and they give different results? Moreover, the Bayesian approach is not by making arbitrary decisions, but about stating priors explicitly, with ML there is nothing explicit, you're using a black box and you have no theoretical foundations for using it this way (as compared to Bayesians). $\endgroup$
    – Tim
    Feb 10, 2018 at 17:28
  • 1
    $\begingroup$ @ThalesWest OLS is just an algorithm for finding parameters of linear regression that is efficient for small to moderately large data. There's nothing "statistical" about it, other algorithms are not more "ML". OLS won't work for huge datasets, unless you also have a huge computer. $\endgroup$
    – Tim
    Mar 3, 2019 at 6:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.