10
$\begingroup$

I want to do multiple linear regression and then to predict new values with little extrapolation. I have my response variable in the range from -2 to +7, and three predictors (the ranges about +10 - +200). The distribution is nearly normal. But the relationship between the response and the predictors is not linear, I see curves on the plots. For example like this: http://cs10418.userapi.com/u17020874/153949434/x_9898cf38.jpg

I would like to apply a transformation to achieve linearity. I tried to transform the response variable by checking different functions and looking at the resulting plots to see a linear relationship between the response and predictors. And I found that there are many functions which can give me visible linear relationship. For example, functions

$t_1=\log(y+2.5)$

$t_2=\frac{1}{\log(y+5)}$

$t_3=\frac{1}{y+5}$

$t_4=\frac{1}{(y+10)^3}$

$t_5=\frac{1}{(y+3)^\frac{1}{3}}$ etc. give the similar results: http://cs10418.userapi.com/u17020874/153949434/x_06f13dbf.jpg

After I am going to back-transform the predicted values (for $t=\frac{1}{(y+10)^3}$ as $y’=\frac{1}{t^\frac{1}{3}}-10$ and so on). The distributions are more or less similar to normal.

How can I choose the best transformation for my data? Is there a quantitative (and not very complicated) way to evaluate linearity? To prove that selected transformation is the best or to find it automatically if possible.

Or the only way is to do the non-linear multiple regression?

$\endgroup$
  • $\begingroup$ I had a go at improving the formatting of your formulae but may have introduced some mistakes - please check. $\endgroup$ – Peter Ellis Mar 7 '12 at 2:24
  • $\begingroup$ I do not believe you. It is not mathematically possible for $t_1$ through $t_5$ to simultaneously have a linear relationship with a sixth variable over the range $0\ldots 200$. I think you may have made a mistake in computing these transformations of $y$. $\endgroup$ – whuber Mar 7 '12 at 3:37
  • 1
    $\begingroup$ @whuber Thank you for the answer. I made the plots in R cs9579.userapi.com/u17020874/153949434/z_9fa17c02.jpg cs9579.userapi.com/u17020874/153949434/z_7fa6891c.jpg $\endgroup$ – nadya Mar 7 '12 at 23:51
  • 2
    $\begingroup$ You're right. That's pretty amazing that such a wide range of re-expressions of y would remain in a linear relationship with r. Thanks for sharing that. If you plot the residuals, you will find that $1/(y+5)$ looks about the best, and then $r$ needs no re-expression: plot(lm(1/(y+5)~r)). $\endgroup$ – whuber Mar 8 '12 at 0:09
14
$\begingroup$

This is somewhat of an art, but there are some standard, straightforward things one can always attempt.

The first thing to do is re-express the dependent variable ($y$) to make the residuals normal. That's not really applicable in this example, where the points appear to fall along a smooth nonlinear curve with very little scatter. So we proceed to the next step.

The next thing is to re-express the independent variable ($r$) to linearize the relationship. There is a simple, easy way to do this. Pick three representative points along the curve, preferably at both ends and the middle. From the first figure I read off the ordered pairs $(r,y)$ = $(10,7)$, $(90,0)$, and $(180,-2)$. Without any information other than that $r$ appears always to be positive, a good choice is to explore the Box-Cox transformations $r \to (r^p-1)/p$ for various powers $p$, usually chosen to be multiples of $1/2$ or $1/3$ and typically between $-1$ and $1$. (The limiting value as $p$ approaches $0$ is $\log(r)$.) This transformation will create an approximate linear relationship provided the slope between the first two points equals the slope between the second pair.

For example, the slopes of the untransformed data are $(0-7)/(90-10)$ = -$0.088$ and $(-2-0)/(180-90)$ = $-0.022$. These are quite different: one is about four times the other. Trying $p=-1/2$ gives slopes of $(0-7)/(\frac{90^{-1/2}-1}{-1/2}-\frac{10^{-1/2}-1}{-1/2})$, etc., which work out to $-16.6$ and $-32.4$: now one of them is only twice the other, which is an improvement. Continuing in this fashion (a spreadsheet is convenient), I find that $p \approx 0$ works well: the slopes are now $-7.3$ and $-6.6$, almost the same value. Consequently, you should try a model of the form $y = \alpha + \beta \log(r)$. Then repeat: fit a line, examine the residuals, identify a transformation of $y$ to make them approximately symmetric, and iterate.

John Tukey provides details and many examples in his classic book Exploratory Data Analysis (Addison-Wesley, 1977). He gives similar (but slightly more involved) procedures to identify variance-stabilizing transformations of $y$. One sample dataset he supplies as an exercise concerns century-old data about mercury vapor pressures measured at various temperatures. Following this procedure enables one to rediscover the Clausius-Clapeyron relation; the residuals to the final fit can be interpreted in terms of quantum-mechanical effects occurring at atomic distances!

$\endgroup$
  • $\begingroup$ Thank you for the advice of the Box-Cox transformation. Does it make any sense to check R-squared of lm(1/(y+5)~r) and lm of other functions and then to compare these R-squared? $\endgroup$ – nadya Mar 8 '12 at 4:36
  • $\begingroup$ It makes sense when r is fixed, because then $R^2$ is a proxy for the variance of the residuals. If you're re-expressing r (the independent variable), though, then $R^2$ is worthless or misleading: see stats.stackexchange.com/questions/13314/…. $\endgroup$ – whuber Mar 8 '12 at 13:14
  • $\begingroup$ Many thanks for the answering! I'm not going to transform my independent variables $\endgroup$ – nadya Mar 8 '12 at 23:16
  • $\begingroup$ @whuber: Assuming I only have a single variable $y$, what is a good rule of thumb to choose the transformation? I have the EDA book by Tukey, but I find it hard to find my way around. A lot seems to be focused on pen-and-paper re-expression. Any page/chapter you find very valueable? $\endgroup$ – Erich Schubert Aug 13 '13 at 8:45
  • $\begingroup$ @Erich Every bit of that book is deeply rewarding: after all, if you can do something with pencil and paper, then you can program a computer to do it :-). With a single variable often it's nice to transform it for symmetry (of its empirical distribution); Tukey calls this a "little deal." A simple way to identify such a transformation is described in section 3E, "Looking quickly." It illustrates what can be learned from a glance at an N-letter summary (Tukey suggests a 7- or 9-letter summary). Acquiring that skill is more valuable than having a computer program do the calculations for you. $\endgroup$ – whuber Aug 13 '13 at 14:08
1
$\begingroup$

If your response variable (or rather, what will become the residuals of your response variable) on the original scale has a Normal distribution as you imply, then transforming it to create a linear relationship with the other variables will mean that it no longer is Normal and it will also change the relationship between its variance and mean values. So from that part of your description I think you are better off using non-linear regression than transforming the response. Otherwise, after linear transformation of the response, you will need a more complex error structure (although this can be a matter of judgement and you would need to check, using graphical methods).

Alternatively, investigate transformation of the explanatory variables. As well as straight transformations, you also have the option of adding in quadratic terms.

More generally, transformation is more an art than a science, if there is no existing theory to suggest what you should use as the basis of transformation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.