0
$\begingroup$

I am building a linear regression model using Ridge regression. Some of the independent variables don't have linear relationships with the dependent variable. I've tried to do data transformations on those variables, but there isn't really a clean transformation to make some of them linear. Would it be a bad idea to drop those variables from my model altogether?

$\endgroup$
4
$\begingroup$

Dropping a predictor just because it doesn't show a linear relation with the response when considered alone is usually a bad idea, because that predictor may be useful when used with other predictors.

I try to show it with an example:

n <- 1e4
x <- rnorm(n)
y <- rnorm(n)
z <- x-y

Let's try to predict variable x using variables y and z as predictors. We can see that variables x and y are independent - neither a linear relation nor any other kind of relation. Correlation is just 0:

data <- data.frame(x,y,z)
cor(data)

             x            y          z
x  1.000000000 -0.009880608  0.7116068
y -0.009880608  1.000000000 -0.7095747
z  0.711606819 -0.709574733  1.0000000

But predicting x from y and z is a perfect fit:

library(faraway)
sumary(lm(x~y+z))

              Estimate Std. Error    t value Pr(>|t|)
(Intercept) 5.4956e-17 5.2156e-17 1.0537e+00   0.2921
y           1.0000e+00 7.3892e-17 1.3533e+16   <2e-16
z           1.0000e+00 5.1917e-17 1.9261e+16   <2e-16

n = 10000, p = 3, Residual SE = 0.00000, R-Squared = 1

However, if we drop predictor y just because it has no linear relationship with response x, we get a much worse fit:

sumary(lm(x~z))

              Estimate Std. Error  t value Pr(>|t|)
(Intercept) -0.0023683  0.0070591  -0.3355   0.7373
z            0.5014441  0.0049513 101.2750   <2e-16

n = 10000, p = 2, Residual SE = 0.70587, R-Squared = 0.51

In summary, in linear regression we should not drop a predictor just because it doesn't have any linear relationship with the response. If you are interested on how to decide when variables should be included or dropped in a model, I suggest reading about variable selection, which is an interesting topic.

$\endgroup$
0
$\begingroup$

Dropping a variable just because it doesn't have linear relationship with response variable is a bad idea. You must understand the variables relationship using scatter plots but not drop it just because it is not linear with the response variable. You should first build, understand the relationship, check if the variables are significant or not and then decide if you want to remove the variables or not.

$\endgroup$
-2
$\begingroup$

In principle, yes! If there is no linear relationship between your predictor and your target, then there is no point in including your predictor into the model, as it will be one more parameter to estimate when you already know it is 0

We must differenciate, however, between there being no linear relationship, and there being a non-linear relationship. For example, if $Y:=e^X$, the relationship between $Y$ and $X$ is non-linear, but you should definitely still include X in your linear model, as larger X still implies larger Y and although the predictions won't be great, they will be better than if you just assume $Y$ is a constant.

So, my advice is to try reasonably simple transformations when possible (I would not go for something too fancy) Also, fortunatelly, there is plenty of software that can help you with model selection. For example, you could try to start by the most complex model (including all of your variables) and then go for an AIC-based stepwise model optimization, where "useless" predictors will tend to be removed.

Finally, understand how to validate your model! As a general piece of advice, I would go for the simplest model that predicts acceptable results and can stand the test of model validation (mainly, the residuals look like "noise", showing no recogniseable patterns in expectation nor variance)

$\endgroup$
  • 1
    $\begingroup$ The first paragraph is plainly wrong, but the second one adds an interesting point to the other answers. I suggest removing some parts of the answer. $\endgroup$ – Pere Jun 11 '19 at 7:48
  • $\begingroup$ @Pere No, it's not wrong. If $X and Y$ are uncorrelated variables, adding $X$ to the prediction of $Y$ will only bring multicolinearity effects. Models should be kept as simple as possible, so there is no point in adding extra-noise. I understand the example you gave, and it is great for a textbook, you real-world data hardly ever behaives like that $\endgroup$ – David Jun 11 '19 at 7:55
  • 1
    $\begingroup$ Please see the example in my answer. X and Y are uncorrelated but X=Y+Z (exactly, R2=1) while Z just gives R2=0.5. X and Y being uncorrelated doesn't mean than Y can improve the predictive value of a set of variables when included in it. $\endgroup$ – Pere Jun 11 '19 at 8:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.