Importance of assumptions in linear regression I have been in conflict with my study materials. I began learning about linear regression from a stats professor who emphasised on fulfilling the 4 assumptions on errors when performing linear regression. This also meant that in-sample performance measures such as AIC,BIC and R2 are mostly used to estimate model performance.
As i begun to delve more into statistical learning and machine learning, I realised that the materials stopped focusing on all of these assumptions, because out-of-sample performance is the most important criteria of model performance. Ideas like regularisation and cross-validation further emphasis the importance of out-of-sample performance.
I am now attempting to reconcile the need to even look at assumptions, and determine when exactly do they matter. As a data analyst, I am more concerned with model inference (which variables affect Y most), as compared to predictions.
I hope to hear some advice on how I can possibly reconcile these two school of thoughts, or if there's a need to take a side.
 A: Say we have two variables for a sample from a population (eg, height and shoe size).  Two approches with different objectives:
Estimate population parameter


*

*We think that these data reflect some truth about the relationship
between height and shoe size--there is some true parameter that
exists--and we want to find it.  That truth might be approximated
with a coefficient from the sample, but the usefulness of that
estimate depends a lot on its variance and bias, whose closed forms
require certain assumptions to hold.  This aim is actually very
powerful because if it is attained we know something about reality. 
In the ideal, ideal case we actually do not even need out of sample
validation because the relationship is clear.  Computers and large
datasets however allow easy out of sample validation, soI think
generally this is also performed as a sanity check--not as the goal. 
You can imagine however that this approach depends a lot on the
analyst, which can be good or bad.


Build a tool


*

*We want to find some mapping that gives us best out of sample
performance. That mapping can be a coefficient---hence it is often
called a "model", but that is an abuse of the term.  We do not really
posit some population parameter and then try to estimate it, we just
require some tool to map.  It is a model in that it is a reduction of
the data, like a model car is a model, but not in the way above where
structure is imposed on the data. The big issue is that even with out
of sample validation, it is hard to truly validate this tool because,
unlike in the first case, we don't know whether it reflects some
population truth or not.  We also have bias and variance, but it is
of the out of sample error now, not of the estimate of the
coefficient, and these quantities are approximated now, unlike in the
first case where they were closed form based on structure we had
imposed---via assumptions---on the structure of the data and
underlying population paramter.  The benefit of this approach is that
the mapping can be as flexible as possible and ---at the risk of
overfitting---we can find something that gives better out of sample
error than with the first approach, where in some ways is constrained
to a model and assumptions you understand.  However, I do not
personally see how this tool could ever truly be validated, and hence
the reticence of using such a tool in some areas such as criminal
justice and medicine.  Interestingly, the general public appears to
not really care that these tools are not validate-able, perhaps
because these tools often fall under the guise of "AI".  Either way,
this approach depends a lot on the data, which can be good or bad.


You can use the optimization portion of "linear regression" for either of these aims.  You are just projecting the response onto the column space of the predictors or equivalently minimizing the MSE.  However the context in which this procedure is performed is totally different in the two cases.
