2
$\begingroup$

I was creating a linear regression model on a particle collisions dataset. I observed that my model breaks several assumptions in linear regression, and when I tried to fix them, it increased the error. Why might this be? I made use of a train-test split during my experiments and here is my methodology using multiple linear regression with no regularisation.

I altered my experiment in 3 stages and each stage increased the mean squared error in my predictions. I plotted the residuals, the standard errors in the coefficients, and the residuals on a histogram. I recorded the mean squared error in predictions on my test set.

These are the result of my original model, with a mean squared error of 388.573.

Original Model

I observed that the residuals are really heavily skewed positively, so I decided to remove outliers in my data that are more than 2 standard deviations from the mean in any predictor column. The aim here was to make my residuals more normally distributed. Here is the results.

Normalising residuals

This increased the mean squared error of my predictions to 426.216 - a huge jump.

Lastly, since some of my coefficients had huge standard errors, I made attempts to remove and eliminate multicollinearity in my model by removing correlated features. This gave the following results.

Eliminating Multicollinearity

This increased my mean squared error to 426.384.

This is really not what I expected.

I would have expected the predictions of my model to improve as I made alterations to meet the Gauss-Markov assumptions in linear regression.

What could explain these outcomes? What might have caused my error to increase?

$\endgroup$
4
  • 3
    $\begingroup$ The Gauss-Markov theorem assumes nothing about normality or feature correlation. $\endgroup$
    – Dave
    Commented Jul 21, 2022 at 15:06
  • $\begingroup$ +1 to @Dave. You don't need normality of residuals. In addition, Gauss-Markov says nothing about the MSE of predictions. Also, I don't think your predictor SEs are in any way "huge". If all you care about is prediction accuracy, go with your initial model, but be aware there is something called "overfitting to the test set". $\endgroup$ Commented Jul 21, 2022 at 15:17
  • 1
    $\begingroup$ What's the nature of the outcome variable that you're trying to model in the regression? Sometimes a transformation (e.g., log) of the outcome or a generalized linear model will provide better performance, depending on the type of outcome. $\endgroup$
    – EdM
    Commented Jul 21, 2022 at 19:36
  • 2
    $\begingroup$ I decided to remove outliers in my data that are more than 2 standard deviations from the mean in any predictor column." I think this alone could very well cause the reduction in performance you see (even if it is not the cause, this is a bad idea). Outliers should not be removed unless they are clear data errors. Otherwise, you are simply biasing the dataset - and since you are doing a train-test split, it is not surprising that the performance on the test data goes down. $\endgroup$
    – mkt
    Commented Jul 22, 2022 at 13:58

1 Answer 1

1
$\begingroup$

I decided to remove outliers in my data that are more than 2 standard deviations from the mean in any predictor column.

I think this alone could very well cause the reduction in performance you see. Outliers should not be removed unless they are clear data errors. Otherwise, you are simply biasing the dataset - and since you are doing a train-test split, it is not surprising that the performance on the test data goes down.

I would add that even if it is not the main cause of this particular problem for you, removing outliers is a bad idea. It's especially poor practice when you are removing what seems to be a large proportion of the data and not just one or two extreme values.

Finally, 'making your linear model meet assumptions' should mean changing your model to address the complexity of the data, NOT changing your data to meet the simplicity of your model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.