I have been in conflict with my study materials. I began learning about linear regression from a stats professor who emphasised on fulfilling the 4 assumptions on errors when performing linear regression. This also meant that in-sample performance measures such as AIC,BIC and R2 are mostly used to estimate model performance.

As i begun to delve more into statistical learning and machine learning, I realised that the materials stopped focusing on all of these assumptions, because out-of-sample performance is the most important criteria of model performance. Ideas like regularisation and cross-validation further emphasis the importance of out-of-sample performance.

I am now attempting to reconcile the need to even look at assumptions, and determine when exactly do they matter. As a data analyst, I am more concerned with model inference (which variables affect Y most), as compared to predictions.

I hope to hear some advice on how I can possibly reconcile these two school of thoughts, or if there's a need to take a side.


Say we have two variables for a sample from a population (eg, height and shoe size). Two approches with different objectives:

Estimate population parameter

  • We think that these data reflect some truth about the relationship between height and shoe size--there is some true parameter that exists--and we want to find it. That truth might be approximated with a coefficient from the sample, but the usefulness of that estimate depends a lot on its variance and bias, whose closed forms require certain assumptions to hold. This aim is actually very powerful because if it is attained we know something about reality. In the ideal, ideal case we actually do not even need out of sample validation because the relationship is clear. Computers and large datasets however allow easy out of sample validation, soI think generally this is also performed as a sanity check--not as the goal. You can imagine however that this approach depends a lot on the analyst, which can be good or bad.

Build a tool

  • We want to find some mapping that gives us best out of sample performance. That mapping can be a coefficient---hence it is often called a "model", but that is an abuse of the term. We do not really posit some population parameter and then try to estimate it, we just require some tool to map. It is a model in that it is a reduction of the data, like a model car is a model, but not in the way above where structure is imposed on the data. The big issue is that even with out of sample validation, it is hard to truly validate this tool because, unlike in the first case, we don't know whether it reflects some population truth or not. We also have bias and variance, but it is of the out of sample error now, not of the estimate of the coefficient, and these quantities are approximated now, unlike in the first case where they were closed form based on structure we had imposed---via assumptions---on the structure of the data and underlying population paramter. The benefit of this approach is that the mapping can be as flexible as possible and ---at the risk of overfitting---we can find something that gives better out of sample error than with the first approach, where in some ways is constrained to a model and assumptions you understand. However, I do not personally see how this tool could ever truly be validated, and hence the reticence of using such a tool in some areas such as criminal justice and medicine. Interestingly, the general public appears to not really care that these tools are not validate-able, perhaps because these tools often fall under the guise of "AI". Either way, this approach depends a lot on the data, which can be good or bad.

You can use the optimization portion of "linear regression" for either of these aims. You are just projecting the response onto the column space of the predictors or equivalently minimizing the MSE. However the context in which this procedure is performed is totally different in the two cases.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.