2
$\begingroup$

Say I have a bunch of explanatory variables to predict a continuous independent variable. Below, a simple toy example:

enter image description here

I think it would be easiest to do a log-log transform and proceed with linear regression. Looks like that the explanatory variables are relatively highly correlated, and there might be a high collinearity between them. This, I would take care of later via e.g.,

  • Lasso regression (or Ridge)
  • feature selection algos
  • Partial Least Squares
  • Decision trees and feature importance
  • Dimensionality reduction via principal component analysis

But back to the log-log-transform; now, the data looks like this:

enter image description here

To me, it looks like that x2, x3, and x4 are now better suited for a linear regression. However, x1 does not look "very linear" anymore. How would I best deal with x1 before I proceed?

$\endgroup$
1
$\begingroup$

I'm not sure what your plots are of since the axes are not properly labeled. However, if your goal is simply to predict $y$ and get the best predictions are possible, and there is no need to interpret the coefficients, why not simply fit the the model with all your variables and then validate the data with a holdout sample. Select the one with the smallest predicted error. Don't worry about your model assumptions so much if your only goal is to predict. It would be good to get a better idea of exactly what you want to do with your model, however, before a "correct" answer could be given to your question.

$\endgroup$
  • $\begingroup$ Thanks for the suggestions. What I want to do with my model is to predict the independent variable (here on the Y axis). I would go even further and to k-fold crossfold validation instead of 1-holdout sample to avoid overfitting. But the question is how to make it a linear problem -- if I use the data as is (like in the upper panel) I would surely get a worse MSE or R^2 compared to the log-log transform. So, here it makes sense to do the transform if I want to use a linear model, but the question is how to deal with mixed variables (linear and non-linear data) $\endgroup$ – user39663 Jan 31 '15 at 6:18
  • $\begingroup$ You could try using a quadratic term for $x_1$ (i.e. $x_1^2$). $\endgroup$ – StatsStudent Jan 31 '15 at 6:21
  • $\begingroup$ BTW, $R^2$ can be improved by simply adding additional terms to your model. $\endgroup$ – StatsStudent Jan 31 '15 at 6:27
  • 1
    $\begingroup$ Agree, I think this was a weird question, but thanks for your help! $\endgroup$ – user39663 Jan 31 '15 at 6:29
1
$\begingroup$

Actually the first plot didn't quite look linear in $x_1$ to me (though it's hard to tell from marginal plots anyway; you would get better information from partial plots).

If the relationship is not linear, you could consider either transforming $x_1$ (particularly in instances where that has a nice interpretation), or fitting a nonlinear form of relationship in $x_1$.

There's too little detail in your question to get much more specific, but the choices of transformation or nonlinear relationship would be better informed by subject-matter knowledge in any case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy