3
$\begingroup$

When performing linear regression, why might you choose to transform one of your predictor variables over using polynomial regression? Essentially what are the advantages (if any) of performing a transformation over using polynomial regression?

As an example I'm using the Auto dataset provided by ISLR2 package:

require(ISLR2)

df <- Auto
df$horsepower <- log(Auto$horsepower, 2)

model.tran <- lm(mpg ~ horsepower, df)
model.poly <- lm(mpg ~ poly(horsepower, 2), Auto)

summary(model.tran)
summary(model.poly)

ggplot(Auto, aes(horsepower, mpg))+
  geom_point()+
  geom_line(aes(y=predict(model.tran, df)), col="blue")+
  ggtitle("Log Transformed")

ggplot(Auto, aes(horsepower, mpg))+
  geom_point()+
  geom_line(aes(y=predict(model.poly, Auto)), col="blue")+
  ggtitle("Polynomial 2 degrees")

output:

Call:
lm(formula = mpg ~ horsepower + horsepower, data = df)

Residuals:
     Min       1Q   Median       3Q      Max 
-14.2299  -2.7818  -0.2322   2.6661  15.4695 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 108.6997     3.0496   35.64   <2e-16 ***
horsepower  -12.8802     0.4595  -28.03   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 4.501 on 390 degrees of freedom
Multiple R-squared:  0.6683,    Adjusted R-squared:  0.6675 
F-statistic: 785.9 on 1 and 390 DF,  p-value: < 2.2e-16


Call:
lm(formula = mpg ~ poly(horsepower, 2), data = Auto)

Residuals:
     Min       1Q   Median       3Q      Max 
-14.7135  -2.5943  -0.0859   2.2868  15.8961 

Coefficients:
                      Estimate Std. Error t value Pr(>|t|)    
(Intercept)            23.4459     0.2209  106.13   <2e-16 ***
poly(horsepower, 2)1 -120.1377     4.3739  -27.47   <2e-16 ***
poly(horsepower, 2)2   44.0895     4.3739   10.08   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 4.374 on 389 degrees of freedom
Multiple R-squared:  0.6876,    Adjusted R-squared:  0.686 
F-statistic:   428 on 2 and 389 DF,  p-value: < 2.2e-16

Plot of model functions onto scatter graph

As can be seen, the R squared value is higher in the polynomial model. The residual standard error is lower. And residual vs fitted plots indicate residuals variance is slightly more constant in the polynomial regression.

This Answer here https://stats.stackexchange.com/a/287475/353359 mentions implications for the model outside of the x-range and variation from the model. Is this simply a risk of overfitting?

While I have included an example, my question is more general. When would you want to use transformations if polynomial regressions seems like a catch-all for non-linearity.

$\endgroup$
4
  • 9
    $\begingroup$ It is slightly implausible the fuel efficiency increase for very high horsepower $\endgroup$
    – Henry
    Mar 20, 2022 at 2:34
  • $\begingroup$ @Henry Does that mean then that by fitting the polynomial regression, the model will include an erroneous bias. And that the transformation is more appropriate in this particular instance, but may not be for other datasets? $\endgroup$ Mar 20, 2022 at 2:48
  • 1
    $\begingroup$ A specific and also general point is that transforming the response can be another option, It's often been commentedm for this kind of data that the reciprocal of miles per gallon, namely gallons per mile, or trivially some multiple of it, can help in modelling; (Those used to some variation on litres per km will be familiar with the idea any way.) $\endgroup$
    – Nick Cox
    Mar 20, 2022 at 16:40
  • $\begingroup$ The different options have different models of the variance with respect to the curve (i.e., the error model). They essentially make a Gaussian/normal error model for different variables. $\endgroup$ Mar 21, 2022 at 6:36

4 Answers 4

15
$\begingroup$

In theory, the implicit assumption in regression is that you know the shape of the function---linear, quadratic, logarithmic etc.---and just need to find its parameters by fitting it to the data. In practice, of course, this is seldom the case. You can fit an infinite number of functions and obtain marvellous results on your training data, but such models will likely perform poorly in the real world.

So, when choosing the model, nothing beats domain knowledge. In case of your dataset, an automotive engineer or a physicist might be in the position to suggest a realistic model. But, even with a cursory knowledge of physics we might be able to exclude many candidates. For example:

  • Can fuel efficiency of a car ever be negative?
  • Is it plausible for the efficiency to rise with the engine power (hat tip to @Henry)?
  • Can it rise infinitely with any change of power?

etc.

The first two points obviously speak against a polynomial model, including a linear one ("linear" refering to predictors). The third speaks against $1/x$, which otherwise would perhaps seem plausible.

Among the infinite number of the functions passing the above plausibility check, the exponential decrease is probably the simplest model you can use.

$\endgroup$
8
  • 1
    $\begingroup$ So I understand that you should theoretically know the shape of your regression function. I also understand there are probably better ways of determining a model through exploratory analysis than linear regression in that case. But theoretically if one was to use linear regression explorativly as I've done here, could you use a validation dataset to assess the models? Also if you found my question to be clear and well written consider upvoting to allow me to comment and learn more elsewhere. $\endgroup$ Mar 20, 2022 at 9:41
  • 1
    $\begingroup$ @JoeBlackSci It is possible and it's being done in practice, but it is a backup option. Domain knowledge is always better. $\endgroup$
    – Igor F.
    Mar 20, 2022 at 9:46
  • 2
    $\begingroup$ The third point is not usually relevant: it makes little sense to insist that a model be accurate for the entire mathematical range of possibilities. Such a requirement would rule out the use of any principle of Newtonian physics, for instance. The first point is helpful but ought to be modified by referring to the measured fuel efficiency, which could be negative. $\endgroup$
    – whuber
    Mar 20, 2022 at 14:15
  • $\begingroup$ Domain knowledge is valuable but always better? This example is from mechanics, so presumably there are physical laws that determine mpg and they can be reasonably approximated as a linear function of the measured predictors. In other domains it might be a lot less obvious how to express knowledge in terms of simple mathematical functions. Maybe sometimes deferring to experts would be a way to introduce bias. $\endgroup$
    – dipetkov
    Mar 20, 2022 at 16:25
  • $\begingroup$ @whuber My third point refers to the physical range of possibilities. I believe this is clear from the consecutive paragraph, referring to $1/x$. Regarding the 1st point, I guess it's useful to distinguish between the physical model and the errors. In the above case, an asymptotically falling, always positive curve is likely a reasonable physical model, but I see that the log-transformation of the fuel efficiency might fail due to negative observed values. $\endgroup$
    – Igor F.
    Mar 21, 2022 at 8:35
9
$\begingroup$

That cloud of data point does not really have a clear shape. There are all sorts of lines that you can draw through it.

different fits

  • You should ideally use a function that makes physical sense. This could be an exponential function but also a power-law could make sense. Ideally, you derive some function theoretically (see below).

  • The polynomial fit is doing well because it is a flexible function.

    If you keep adding terms you can make the curve more flexible and the fit even better. But the data points do not have a spectacular shape and it only deviates a little bit from a straight line. So a second-order polynomial is already doing very well.

    So yes, a polynomial can fit this cloud of points. The reason is that the cloud of points has no difficult shape and almost any function that creates a curve with a slight bend can fit the points.

    The downside of a polynomial is that it is only a phenomenological model. It doesn't relate to any mechanism and the parameters have no meaning in a mechanistic sense. If the shape of points is more complex (e.g. if you have a larger range for the parameters and more change in the curvature) then polynomial may perform less well than some function that is derived based on some mechanistic principles (also note that if you plot the residuals then you see that they are not homogeneous and so the polynomial probably has some bias relative to the 'true model').

  • Your log-transform fit is performing the worst

    In your case you fit a sort of exponential function

    $$\text{mpg} = a+b \log_2(\text{hp})$$

    which is equivalent to $\frac{-a}{b} + \frac{1}{b}\cdot \text{mpg} = \log_2(\text{hp})$ and could be expressed as $\text{hp}$ being an exponential function of $\text{mpg}$

    $$\text{hp} = e^{c + d \cdot \text{mpg}}$$

    where $c = (\frac{-a}{b}) \log 2$ and $d = (\frac{1}{b}) \log 2$. But to me it is not clear why you would do this. It seems a bit unmotivated and more like random trial and error applying transforms until you find something that fits better.

    However, we can ignore that the log transform makes little sense here and address the question 'why use log transform over polynomial regression' in a more general sense by speaking about log-transform or another function that makes more sense (like the power law 1/x described below). We would use it over polynomial regression because it could relate better to some mechanistic principles that are underlying the data.

    In addition, an advantage for log-transforms can be that it makes it possible to work with data that spans a large range. See for instance the Hertzsprung–Russell diagram

    hp diagram

Potential mechnistic model

The linear function below makes somewhat sense

$$\text{gallons per mile} = \frac{1}{mpg} = \text{constant} \cdot \text{horsepower}$$

The fuel usage in this dataset is for the highway where all cars drive the same speed. Cars with a higher amount of horsepower are probably gonna use more energy. Probably because they are heavier loaded or bigger (if that is why they have the additional horsepower).

If we assume that a car uses all of their horsepower, then for a given time (which is equivalent to 'given distance' if the cars are compared while they all drive the same speed), the energy needs, and the fuel usage, will be proportional to the horsepower.

The plot below on the left shows that a straight line fits reasonably well. On the right we see the fit when the y-scale is not inverted. The $R^2 = 0.643$ is not as good as with the other models like the exponential fits or the power law fits above. But the difference is not that large. The additional parameters that are necessary for the other fit can only increase the explained variance by a few percent.

The advantage of this fit is that the estimated value for the constant of proportionality, 0.0004472, has a physical meaning and sense. Instead of deriving it from data we could also derive it with an on the back of an envelope computation.

  • Say the cars drive 70 mph, then a single mile takes 3600/70 = 51.4 seconds per mile.
  • One horsepower is about 745.7 Watts, then one mile takes 51.4*745.7 = 38350 Joules per mile per horsepower.
  • One Gallon of gasoline contains about 114000 BTU per gallon (and one BTU is about 1055 Joules) so, one mile takes 38350/(114000*1055)= 0.00031 Gallons per mile per horsepower.
  • It takes a bit more than that because of engine efficiency and not all energy from fuel is converted to mechanical energy/work/power. Say if the efficiency is 30% then the energy usage is 0.00031/0.3 = 0.00106 gallons per mile per horsepower.

This constant of 0.00106 is about twice as large as what we see in the data. Cars below the 0.00106 line are using less fuel and are probably not needing/using all their horsepower on the highway. Cars above the 0.00106 line are using more fuel and are probably less efficient with their fuel (less mechanic energy/power for the same amount of energy from fuel).

linear

$\endgroup$
4
  • $\begingroup$ Thank you for answer it's given me a lot to think about. I have a follow up question. When you say there could be better functions that relate hp with mpg do you mean another form of transformation, or including other predictors in multiple linear regression? E.g. displacement or car weight that could be responsible for the variation seen at low horsepower values. $\endgroup$ Mar 20, 2022 at 11:54
  • 1
    $\begingroup$ @JoeBlackSci Better functions are sometimes non-linear in the parameters and do not always have a known linearizing transform. $\endgroup$
    – Galen
    Mar 20, 2022 at 15:18
  • 1
    $\begingroup$ +1. In the scope and depth of its concepts, the utility of its advice, the richness of the examples, and the detail of its calculations, this answer is far superior to the accepted one. $\endgroup$
    – whuber
    Mar 21, 2022 at 13:49
  • $\begingroup$ I agree with @whuber. ISTR that gallons per mile is considered a more interpretable representation of a vehicles economy than miles per gallon when human beings are reasoning about it - it looks to be the same for statistical reasoning! reuters.com/article/us-fuel-efficiency-idUSN1925607520080619 $\endgroup$ Mar 21, 2022 at 22:18
7
$\begingroup$

There are several problems with the use of simple transformations such as log, e.g.

  • How do you handle zeros or negative values that occur in some data?
  • Simple transformations are often better than not transforming but not as good as more flexible multiparameter fits (e.g., spline functions, fractional polynomials)
  • One seldom knows how to pre-specify a single transformation and hence looks at the performance of multiple possible transformations. This creates a major bias in final standard errors and "significance" tests due to model uncertainty.

Regression splines are flexible and fully expose the proper number of degrees of freedom to use in estimating standard errors and performing statistical tests.

$\endgroup$
5
  • 3
    $\begingroup$ Generally on point, but in this particular case we know that always y=mpg > 0 and thus the deterministic log transform is a useful way to a) address heteroscedasticity; b) turn values into whole real line ( which then are amenable for splines , nonparametric function fits , whicb are often unconstrained by default). $\endgroup$ Mar 20, 2022 at 12:52
  • 3
    $\begingroup$ @Georg Although the underlying $y$ might be positive, its measurements could be negative. For instance, suppose a car's fuel volume were measured; the car is driven a short distance; and the fuel volume re-measured. If those measurements are independent and subject to Normally distributed error, potentially the measured fuel consumption is negative, whence the computed mileage $y$ is negative, too. Such considerations help us better appreciate the points Prof. Harrell raises here. $\endgroup$
    – whuber
    Mar 20, 2022 at 14:11
  • 2
    $\begingroup$ @whuber (+1) the (possibly faulty) range meter on my car occasionally does exactly that (i.e. the "miles left" goes up during the journey rather than down). I think it is due to miscalibration rather than random measurement error as it tends to vary as a function of the amount of petrol in the tank (I collected data for statistical modelling ;o). $\endgroup$ Mar 20, 2022 at 17:57
  • $\begingroup$ I followed the model uncertainty link and don't have access to the full text. Am I right in thinking that by performing exploratory data analysis without considering the mechanisms behind the data that you are essentially just choosing a model that works well for your data, and you are making assumptions that may not be true to the "true function" of the model and hence introducing error through model bias? $\endgroup$ Mar 21, 2022 at 0:29
  • $\begingroup$ Yes, and model uncertainty makes estimates of precision far too optimistic. $\endgroup$ Mar 21, 2022 at 18:35
4
$\begingroup$

@IgorF. and @SextusEmpiricus (+1) make the key point "You should ideally use a function that makes physical sense.", a nice example is allometry which is often used in biology to model the relationships between physical measurements, for instance the long bone circumference and body mass of dinosaurs,

G. C. Cawley and G. J. Janacek, On allometric equations for predicting body mass of dinosaurs, Journal of Zoology, vol. 280, no. 4, pp. 355-361, 2010. doi:10.1111/j.1469-7998.2009.00665.x (preprint)

which was a comment on an earlier paper by Packard et al, that got a fair bit of media attention as it (incorrectly) asserted that dinosaurs were only about half as big as was previously thought.

G. C. Packard, T. J. Boardman, G. F. Birchard, Allometric equations for predicting body mass of dinosaurs, Journal of Zoology, vol. 279, no. 1, pp. 102-110, 2009. doi:10.1111/j.1469-7998.2009.00594.x

Such relationships often have a power-law relationship, e.g. the strength of a bone depends on its cross-sectional area, and the body mass is related to the cube of the linear body measurements. So we would expect a power-law relationship between the circumference of major bones, such as the femur, and the body mass of the dinosaur. In practice it won't be exactly cubic, but allometric models aim to determine the appropriate scaling from the observations. If we use logarithmic transformations of both variables, then a linear model gives a power-law relationship, so that implements our prior knowledge about the problem. As a bonus, it also imposes an assumption of relative rather than absolute errors, which turns out to be the other key benefit of allometry over other transformations. Our paper was a comment on a previous study that, err..., did not fully appreciate that.

This example shows some advantages of log transformation in allometry, (a) and (b) show the long bone circumferences and body masses of 33 mammals (used to predict quadrapedal dinosaur body masses), plotted on log-log axes and on linear axes. This shows that a linear model on log-log transformed data is a very reasonable model. (c) and (d) show the power law model of Packard et al. which was fitted using least squares without the log transformation, which looks O.K. (ish) on linear axes, but of you back-transform the model to log-log axes, then it is very clearly biased, with the body masses of smaller mammals being consistently over-predicted. The predictive error bars of the model also imply that it is possible for some smaller mammals to have a negative body mass.

example from paper

Unfortunately Packard et al. used this model to argue that the body masses of dinosaurs was only about half that previously thought, but this is because their model is very sensitive to the body mass of the largest mammal in the dataset (the elephant) because of the assumption of absolute, rather than relative errors. This means a difference of a kilogram in the weight of an elephant is as bad as an error of a kilogram in the predicted weight of a mouse! The error in the modelling assumptions is easily demonstrated by making predictions with the elephant left out of the calibration data:

enter image description here

The conventional allometric model is just about able to explain the observed body mass of an elephant (it is an outlier among mammals - it has very strong bones for it's body mass, perhaps because it is a very active animal). The model fitted on linear axes on the other hand has the elephant lying many standard deviations away from the predicted value. The conventional model predicts the elephant as having a body mass of about 10 tons (4 would be a more accurate estimate). The Packard et al model though, predicts the mass to be 36 tons, which is wildly wrong (and the estimates for the body masses of dinosaurs are also much higher - the estimated mass of Apatosaurus louisiae rises from 18.2 tons to 302 tons, which is completely implausible). The reason for this is the next most heavy mammal is the hippo, which has a relatively high body mass for its bone circumference.

So this shows that there are two reasons for logarithmic transformation: Firstly for including prior knowledge of the form of the regression; Secondly for including prior knowledge regarding the distribution of values around the regression (which is normally the conditional mean of the predictive distribution).

$\endgroup$
1
  • $\begingroup$ "Our paper was a comment on a previous study that, err..., did not fully appreciate that." I don't follow this sentence. $\endgroup$ Mar 21, 2022 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.