Linear vs. nonlinear regression

Question

I have a set of values $x$ and $y$ which are theoretically related exponentially:

$y = ax^b$

One way to obtain the coefficients is by applying natural logarithms in both sides and fitting a linear model:

> fit <- lm(log(y)~log(x))
> a <- exp(fit$coefficients[1])
> b <- fit$coefficients[2]

Another way to obtain this is using a nonlinear regression, given a theoretical set of start values:

> fit <- nls(y~a*x^b, start=c(a=50, b=1.3))

My tests show better and more theory-related results if I apply the second algorithm. However, I would like to know the statistical meaning and implications of each method.

Which of them is better?

Answer 1

"Better" is a function of your model.

Part of the reason for your confusion is you only wrote half of your model.

When you say $y=ax^b$, that's not actually true. Your observed $y$ values aren't equal to $ax^b$; they have an error component.

For example, the two models you mention (not the only possible models by any means) make entirely different assumptions about the error.

You probably mean something closer to $E(Y|X=x) = ax^b\,$.

But then what do we say about the variation of $Y$ away from that expectation at a given $x$? It matters!

• When you fit the nonlinear least squares model, you're saying that the errors are additive and the standard deviation of the errors is constant across the data:

  $\: y_i \sim N(ax_i^b,\sigma^2)$

  or equivalently

  $\: y_i = ax_i^b + e_i$, with $\text{var}(e_i) = \sigma^2$

• by contrast when you take logs and fit a linear model, you're saying the error is additive on the log scale and (on the log scale) constant across the data. This means that on the scale of the observations, the error term is multiplicative, and so the errors are larger when the expected values are larger:

  $\: y_i \sim \text{logN}(\log a+b\log x_i,\sigma^2)$

  or equivalently

  $\: y_i = ax_i^b \cdot \eta_i$, with $\eta_i \sim \text{logN}(0,\sigma^2)$

  (Note that $\text{E}(\eta)$ is not 1. If $\sigma^2$ is not very small, you will need to allow for this effect if you want a reasonable approximation for the conditional mean of $Y$)

(You can do least squares without assuming normality / lognormal distributions, but the central issue being discussed still applies ... and if you're nowhere near normality, you should probably be considering a different error model anyway)

So what is best depends on which kind of error model describes your circumstances.

[If you're doing some exploratory analysis with some kind of data that's not been seen before, you'd consider questions like "What do your data look like? (i.e. $y$ plotted against $x$? What do the residuals look like against $x$?". On the other hand if variables like these are not uncommon you should already have information about their general behaviour.]

Answer 2

When you fit either model, you are assuming that the set of residuals (discrepancies between the observed and predicted values of Y) follow a Gaussian distribution. If that assumption is true with your raw data (nonlinear regression), then it won't be true for the log-transformed values (linear regression), and vice versa.

Which model is "better"? The one where the assumptions of the model most closely match the data.