Why would you perform transformations over polynomial regression?

Question

When performing linear regression, why might you choose to transform one of your predictor variables over using polynomial regression? Essentially what are the advantages (if any) of performing a transformation over using polynomial regression?

As an example I'm using the Auto dataset provided by ISLR2 package:

require(ISLR2)

df <- Auto
df$horsepower <- log(Auto$horsepower, 2)

model.tran <- lm(mpg ~ horsepower, df)
model.poly <- lm(mpg ~ poly(horsepower, 2), Auto)

summary(model.tran)
summary(model.poly)

ggplot(Auto, aes(horsepower, mpg))+
  geom_point()+
  geom_line(aes(y=predict(model.tran, df)), col="blue")+
  ggtitle("Log Transformed")

ggplot(Auto, aes(horsepower, mpg))+
  geom_point()+
  geom_line(aes(y=predict(model.poly, Auto)), col="blue")+
  ggtitle("Polynomial 2 degrees")

output:

Call:
lm(formula = mpg ~ horsepower + horsepower, data = df)

Residuals:
     Min       1Q   Median       3Q      Max 
-14.2299  -2.7818  -0.2322   2.6661  15.4695 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 108.6997     3.0496   35.64   <2e-16 ***
horsepower  -12.8802     0.4595  -28.03   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 4.501 on 390 degrees of freedom
Multiple R-squared:  0.6683,    Adjusted R-squared:  0.6675 
F-statistic: 785.9 on 1 and 390 DF,  p-value: < 2.2e-16


Call:
lm(formula = mpg ~ poly(horsepower, 2), data = Auto)

Residuals:
     Min       1Q   Median       3Q      Max 
-14.7135  -2.5943  -0.0859   2.2868  15.8961 

Coefficients:
                      Estimate Std. Error t value Pr(>|t|)    
(Intercept)            23.4459     0.2209  106.13   <2e-16 ***
poly(horsepower, 2)1 -120.1377     4.3739  -27.47   <2e-16 ***
poly(horsepower, 2)2   44.0895     4.3739   10.08   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 4.374 on 389 degrees of freedom
Multiple R-squared:  0.6876,    Adjusted R-squared:  0.686 
F-statistic:   428 on 2 and 389 DF,  p-value: < 2.2e-16

As can be seen, the R squared value is higher in the polynomial model. The residual standard error is lower. And residual vs fitted plots indicate residuals variance is slightly more constant in the polynomial regression.

This Answer here https://stats.stackexchange.com/a/287475/353359 mentions implications for the model outside of the x-range and variation from the model. Is this simply a risk of overfitting?

While I have included an example, my question is more general. When would you want to use transformations if polynomial regressions seems like a catch-all for non-linearity.

Answer 1

In theory, the implicit assumption in regression is that you know the shape of the function---linear, quadratic, logarithmic etc.---and just need to find its parameters by fitting it to the data. In practice, of course, this is seldom the case. You can fit an infinite number of functions and obtain marvellous results on your training data, but such models will likely perform poorly in the real world.

So, when choosing the model, nothing beats domain knowledge. In case of your dataset, an automotive engineer or a physicist might be in the position to suggest a realistic model. But, even with a cursory knowledge of physics we might be able to exclude many candidates. For example:

• Can fuel efficiency of a car ever be negative?
• Is it plausible for the efficiency to rise with the engine power (hat tip to @Henry)?
• Can it rise infinitely with any change of power?

etc.

The first two points obviously speak against a polynomial model, including a linear one ("linear" refering to predictors). The third speaks against $1/x$, which otherwise would perhaps seem plausible.

Among the infinite number of the functions passing the above plausibility check, the exponential decrease is probably the simplest model you can use.

Answer 2

That cloud of data point does not really have a clear shape. There are all sorts of lines that you can draw through it.

• You should ideally use a function that makes physical sense. This could be an exponential function but also a power-law could make sense. Ideally, you derive some function theoretically (see below).

• The polynomial fit is doing well because it is a flexible function.

  If you keep adding terms you can make the curve more flexible and the fit even better. But the data points do not have a spectacular shape and it only deviates a little bit from a straight line. So a second-order polynomial is already doing very well.

  So yes, a polynomial can fit this cloud of points. The reason is that the cloud of points has no difficult shape and almost any function that creates a curve with a slight bend can fit the points.

  The downside of a polynomial is that it is only a phenomenological model. It doesn't relate to any mechanism and the parameters have no meaning in a mechanistic sense. If the shape of points is more complex (e.g. if you have a larger range for the parameters and more change in the curvature) then polynomial may perform less well than some function that is derived based on some mechanistic principles (also note that if you plot the residuals then you see that they are not homogeneous and so the polynomial probably has some bias relative to the 'true model').

• Your log-transform fit is performing the worst

  In your case you fit a sort of exponential function

  $$\text{mpg} = a+b \log_2(\text{hp})$$

  which is equivalent to $\frac{-a}{b} + \frac{1}{b}\cdot \text{mpg} = \log_2(\text{hp})$ and could be expressed as $\text{hp}$ being an exponential function of $\text{mpg}$

  $$\text{hp} = e^{c + d \cdot \text{mpg}}$$

  where $c = (\frac{-a}{b}) \log 2$ and $d = (\frac{1}{b}) \log 2$. But to me it is not clear why you would do this. It seems a bit unmotivated and more like random trial and error applying transforms until you find something that fits better.

  However, we can ignore that the log transform makes little sense here and address the question 'why use log transform over polynomial regression' in a more general sense by speaking about log-transform or another function that makes more sense (like the power law 1/x described below). We would use it over polynomial regression because it could relate better to some mechanistic principles that are underlying the data.

  In addition, an advantage for log-transforms can be that it makes it possible to work with data that spans a large range. See for instance the Hertzsprung–Russell diagram

  

Potential mechnistic model

The linear function below makes somewhat sense

$$\text{gallons per mile} = \frac{1}{mpg} = \text{constant} \cdot \text{horsepower}$$

The fuel usage in this dataset is for the highway where all cars drive the same speed. Cars with a higher amount of horsepower are probably gonna use more energy. Probably because they are heavier loaded or bigger (if that is why they have the additional horsepower).

If we assume that a car uses all of their horsepower, then for a given time (which is equivalent to 'given distance' if the cars are compared while they all drive the same speed), the energy needs, and the fuel usage, will be proportional to the horsepower.

The plot below on the left shows that a straight line fits reasonably well. On the right we see the fit when the y-scale is not inverted. The $R^2 = 0.643$ is not as good as with the other models like the exponential fits or the power law fits above. But the difference is not that large. The additional parameters that are necessary for the other fit can only increase the explained variance by a few percent.

The advantage of this fit is that the estimated value for the constant of proportionality, 0.0004472, has a physical meaning and sense. Instead of deriving it from data we could also derive it with an on the back of an envelope computation.

• Say the cars drive 70 mph, then a single mile takes 3600/70 = 51.4 seconds per mile.
• One horsepower is about 745.7 Watts, then one mile takes 51.4*745.7 = 38350 Joules per mile per horsepower.
• One Gallon of gasoline contains about 114000 BTU per gallon (and one BTU is about 1055 Joules) so, one mile takes 38350/(114000*1055)= 0.00031 Gallons per mile per horsepower.
• It takes a bit more than that because of engine efficiency and not all energy from fuel is converted to mechanical energy/work/power. Say if the efficiency is 30% then the energy usage is 0.00031/0.3 = 0.00106 gallons per mile per horsepower.

This constant of 0.00106 is about twice as large as what we see in the data. Cars below the 0.00106 line are using less fuel and are probably not needing/using all their horsepower on the highway. Cars above the 0.00106 line are using more fuel and are probably less efficient with their fuel (less mechanic energy/power for the same amount of energy from fuel).

Answer 3

There are several problems with the use of simple transformations such as log, e.g.

• How do you handle zeros or negative values that occur in some data?
• Simple transformations are often better than not transforming but not as good as more flexible multiparameter fits (e.g., spline functions, fractional polynomials)
• One seldom knows how to pre-specify a single transformation and hence looks at the performance of multiple possible transformations. This creates a major bias in final standard errors and "significance" tests due to model uncertainty.

Regression splines are flexible and fully expose the proper number of degrees of freedom to use in estimating standard errors and performing statistical tests.

Answer 4

@IgorF. and @SextusEmpiricus (+1) make the key point "You should ideally use a function that makes physical sense.", a nice example is allometry which is often used in biology to model the relationships between physical measurements, for instance the long bone circumference and body mass of dinosaurs,

G. C. Cawley and G. J. Janacek, On allometric equations for predicting body mass of dinosaurs, Journal of Zoology, vol. 280, no. 4, pp. 355-361, 2010. doi:10.1111/j.1469-7998.2009.00665.x (preprint)

which was a comment on an earlier paper by Packard et al, that got a fair bit of media attention as it (incorrectly) asserted that dinosaurs were only about half as big as was previously thought.

G. C. Packard, T. J. Boardman, G. F. Birchard, Allometric equations for predicting body mass of dinosaurs, Journal of Zoology, vol. 279, no. 1, pp. 102-110, 2009. doi:10.1111/j.1469-7998.2009.00594.x

Such relationships often have a power-law relationship, e.g. the strength of a bone depends on its cross-sectional area, and the body mass is related to the cube of the linear body measurements. So we would expect a power-law relationship between the circumference of major bones, such as the femur, and the body mass of the dinosaur. In practice it won't be exactly cubic, but allometric models aim to determine the appropriate scaling from the observations. If we use logarithmic transformations of both variables, then a linear model gives a power-law relationship, so that implements our prior knowledge about the problem. As a bonus, it also imposes an assumption of relative rather than absolute errors, which turns out to be the other key benefit of allometry over other transformations. Our paper was a comment on a previous study that, err..., did not fully appreciate that.

This example shows some advantages of log transformation in allometry, (a) and (b) show the long bone circumferences and body masses of 33 mammals (used to predict quadrapedal dinosaur body masses), plotted on log-log axes and on linear axes. This shows that a linear model on log-log transformed data is a very reasonable model. (c) and (d) show the power law model of Packard et al. which was fitted using least squares without the log transformation, which looks O.K. (ish) on linear axes, but of you back-transform the model to log-log axes, then it is very clearly biased, with the body masses of smaller mammals being consistently over-predicted. The predictive error bars of the model also imply that it is possible for some smaller mammals to have a negative body mass.

Unfortunately Packard et al. used this model to argue that the body masses of dinosaurs was only about half that previously thought, but this is because their model is very sensitive to the body mass of the largest mammal in the dataset (the elephant) because of the assumption of absolute, rather than relative errors. This means a difference of a kilogram in the weight of an elephant is as bad as an error of a kilogram in the predicted weight of a mouse! The error in the modelling assumptions is easily demonstrated by making predictions with the elephant left out of the calibration data:

The conventional allometric model is just about able to explain the observed body mass of an elephant (it is an outlier among mammals - it has very strong bones for it's body mass, perhaps because it is a very active animal). The model fitted on linear axes on the other hand has the elephant lying many standard deviations away from the predicted value. The conventional model predicts the elephant as having a body mass of about 10 tons (4 would be a more accurate estimate). The Packard et al model though, predicts the mass to be 36 tons, which is wildly wrong (and the estimates for the body masses of dinosaurs are also much higher - the estimated mass of Apatosaurus louisiae rises from 18.2 tons to 302 tons, which is completely implausible). The reason for this is the next most heavy mammal is the hippo, which has a relatively high body mass for its bone circumference.

So this shows that there are two reasons for logarithmic transformation: Firstly for including prior knowledge of the form of the regression; Secondly for including prior knowledge regarding the distribution of values around the regression (which is normally the conditional mean of the predictive distribution).