# Machine Learning Exercise Linear Regression

This is a school exercise and I just don't get on the right track.

Data description:

• +- 1000 Samples test data.
• 1 Y column.
• 15 X columns.
• Y mean: ~6 max: 341 min: -221
• X: mean of each column around +- 1 (0.9, 1.05, 0.99 etc.). max over all X: 4.7 min over all X: -2.7

The target rsme should be around 15 but I just can't get below 20. I used every tool every library there is. Things I tried with sklearn:
Models: LinearRegression,KernelRegression, KernelRidge, BayesianRidge.
Feature selection: PolynomialFeatures up to degree 5 but everything over 2 mostly the rsme gets worse. from 1 to degree 2 is the most significant reduction in rsme visible.

A few plots:

The one below is sorted for Y

I tried to post more pictures below of the x y relations but I can't here because I'm only allowed to post 2.

I should mention that we had only linear regression already and it should be kinda solvable with that they told.

• This is a question that requires the self-study tag. Mar 30, 2017 at 12:44
• did you try removing variables from the model that are not well correlated with y? did you try adding interaction terms (e.g. x1*x2)? Mar 30, 2017 at 17:27

## 2 Answers

Look closely to the distribution of your data. The mean is around 6 while the max and min values are 341 and -221 respectively. This suggests that there a number of outliers in your dataset and nothing is more worse for a regression problem than outliers. So, you need to remove the outliers first. Then you should standardize the data values by using a scaler. After these, you should try a regression model.

• Hi. Thanks for your answer. I used the Scaler class from sklearn to Scale my X. And for the outliners i trained my model then removed the top 10% of wrongest predicted X and trained again. should i remove the outliners before i train?
– Sim
Mar 30, 2017 at 12:41
• Yes you can remove all the outliers in the way you are doing but I find Tukey's method more reliable. Remove outliers using Tukey's method before training Mar 30, 2017 at 17:14
• @Sim did you end up getting a better result after removing outliers? Apr 3, 2017 at 1:16

Here are some more pictures to the relation between x & y