0
$\begingroup$

If i dont care about the individual effects of each variable onto y, and building confidence intervals and stuff, but only that i get accurate predictions of my y values. Will satisfying them matter? If they do matter, then why are they not taken into consideration when we do linear regression in machine learning? The coeffs of the input variables that are produced remain the same, and if we get bad results when we do not satisfy them, we should surely take them into consideration, right?

$\endgroup$
1

2 Answers 2

1
$\begingroup$

It depends. You can measure prediction quality by cross-validation and the like, and this doesn't require some assumptions such as normality and linearity. If the assumptions are fulfilled, you are in a better position for achieving a good prediction result, however this is not a theorem and applies sometimes but not always (for example a truly linear system with large variance will yield worse OLS predictions than a slightly nonlinear one with a small variance, at least if the nonlinearity is suitable for linear approximation).

However, you may be interested in future prediction in regions in which you currently don't have test data, and then "slight nonlinearity" may play out much worse than you can assess from the data you have.

Furthermore, standard estimation of the prediction error with test and training sets still requires independence and identical distribution (given explanatory variables $x$) of test and training set, and of both regarding any future observations that you actually want to predict. If this is violated and you don't know how exactly, you cannot assess the prediction error in any reliable way.

PS: "If they do matter, then why are they not taken into consideration when we do linear regression in machine learning?" Well, everybody who ignores these things does so at their own peril...

$\endgroup$
4
  • $\begingroup$ If what have independence and identical distributions? $\endgroup$
    – Dave
    May 20, 2020 at 20:49
  • $\begingroup$ What precisely is required depends on what precisely you do for prediction quality assessment, and what your modeling approach is (particularly, do you treat $x$ as random or fixed?). Roughly, if you have test, training, and unobserved future data, all three of these need to be independent (unless you know how exactly they depend on each other) and generated in a manner that allows generalisation from one to another. Under standard linear regression model assumptions all this will hold. $\endgroup$ May 20, 2020 at 21:33
  • $\begingroup$ And what needs to be identical? $\endgroup$
    – Dave
    May 20, 2020 at 21:53
  • $\begingroup$ Didn't I say this already? The relevant distribution in test set, training set, and future observations. $\endgroup$ May 21, 2020 at 13:49
1
$\begingroup$

Yes, but the strength of my Yes varies by the assumption. For instance, one of the assumptions of OLS is that the model is linear: $y=X\beta+\varepsilon$, You should agree with me that if this assumption is broken, then predicting becomes problematic. Also think about autocorrelation: if $E[X\varepsilon]>0$ then larger $X$ would tend to underestimate and predict with bias. etc.

Machine learning is almost entirely an interpolation exercise. You make the network weights remember a lot of combinations of inputs then interpolate between them when you get a new input. Although the formulae can be exactly the same, there is a difference in application. In a typical OLS setup you get some data X, then try to come up with a few coefficients $\beta$, typically much smaller set of them compared to the umber of observations. In ML you often devise much larger set of weights, even larger than the number of observations often. In this case the weights effectively "memorize" the mapping of $y$ to combinations of $X$. The OLS machinery is simple used as a way to memorize this mapping. The trouble comes when new $X'$ comes that is completely out of the field of previous $X$ that the algorithm has seen so far. In this case it's easy to get a nonsensical result.

$\endgroup$
2
  • $\begingroup$ Ok, so you mean to say that i would get bad y values with large errors, right? $\endgroup$ May 20, 2020 at 19:30
  • $\begingroup$ @divyamsureka, I'm saying that you can't simply ignore assumptions even when predicting. Some of them may not be important in certain cases, but you have to look examine them closer each time. $\endgroup$
    – Aksakal
    May 20, 2020 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.