6
$\begingroup$

I saw this question posted on glassdoor for a data scientist position, and it got my thinking. When people use the word performance, do they typically refer to (1) performance on the training data (2) performance of test data (3) a combination of the 2?

If it's (1), I believe the typical metric is $R^2$? $R^2$ is only relevant for training data, and doesn't say anything about how well your model will do on unseen data.

Another metric that I learned in an ML class is simply comparing the MSE of the test data vs. the training data. MSE itself isn't normalized, so it doesn't say much in terms of absoluteness, but relatively (in this case test MSE vs. training MSE), we can perhaps make some insights. If we see a large test MSE relative to training MSE, then perhaps our model is overfitting.

What are some other metrics to assess the "performance" of linear regression?

$\endgroup$
6
$\begingroup$

The question is rather broad, though I guess that is the intention with an interview question :)

How to measure performance of linear regression?

You question then goes on to talk about training and test sets, and of course this could be part of the answer, however I would back up a bit and suggest that performance should really be in terms of whatever question the model was built to answer. If asked this question in a real world setting by someone who had fitted a model and came to me for advice, the first thing I would ask is:

  • What question is the model trying to answer ?

In any applied setting be it traditional statistical analysis, "modern" data science, or whatever, context is vitally important. As a large part of this I would want to know if the model is going to be used for inference or prediction. With inference, there is almost always a underlying causal question: For example, does wearing seatbelts cause road traffic accident casualties to decrease? Does smoking reduce poor outcomes in those who test positive for COVID-19? In such cases one of the most important aspects of a linear regression is variable selection. It is important not to condition on mediators, or to over-adjust for confounders, but the inclusion of competing exposures is helpful.

If there are several questions, as is often the case in data science settings, such as "which of these variables are most important for understanding the outcome" - the usual (mistaken) approach is some stepwise procedure based on p-values, AIC, BIC etc. This is a terrible idea because it completely overlooks the causal relationships and is based on arbitrary thresholds (even with AIC/BIC this really boils down to p-value thresholds). The key point is that, for each variable of interest, a different model is usually needed, because, for example, a variable that is a confounder for one association, will be a mediator for another, and should be excluded in the former and included in the latter. It should be obvious to anyone who reads published papers that this is the "Table 2 Fallacy" where researchers simply put everything in multivariable linear regression models and report all the estimates and p-values!

Once we are happy with the set of variables we would want to look at the usual diagnostics to assess whether the assumption of linearity holds. Other assumptions such as observation independence and variance homogeneity might also be considered.

Then we might want to think about prediction. People often think that prediction and inference are two separate things, and they are, but the problem is that many people who think they are only interested in prediction are actually interested in inference as well. For instance, right now there are teams of data scientists and machine learning practitioners busy trying to apply their methods to COVID-19 datasets (I was one of them until quite recently), and for the most part all the discussion was around prediction. Yet, after you build a model and find that (and I am making this up for confidentiality reasons) vaping is strongly predictive of better outcome in those aged under 35 who test positive for COVID-19, there ALWAYS follows questions from clinicians about interpreting model output as a causal effect - either with a view to promoting some kind of intervention, or to explain an "apparent paradox". But the regression models were not built with causal inference in mind. This type of thing is happening all the time, and it is utterly bonkers.

If there is genuinely no interest in inference then the "blind" methods of train, test, (cross) validate can of course be used and things like MSE are of course good metrics. However, if there is some element of causality involved then it is important to consider the issues I described above.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Sorry about this. I missed the notification when you originally posted this and didn't see it until just now. $\endgroup$ – David Aug 24 at 19:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.