# Regarding linear and nonlinear relationships

If I understand correctly, in case the underlying relationship between variables is nonlinear (slope parameter is not constant), it can sometimes be linearized for modeling purposes.

This has something to do with the type of nonlinearity exhibited by the relationship.

It appears that in case the relationship is nonlinear in variables – it can be linearized (thus, model coefficients can be estimated in the OLS framework).

However, if the relationship is nonlinear in parameters – it cannot be linearized.

Could you please confirm whether this interpretation is correct, and, more importantly, explain why relationships that are nonlinear in parameters cannot, in general, be linearized.

I think you have two issues confused here:

1. Nonlinear relationships between variables can still be modeled with linear regression if the relationship is linear in the model coefficients.

2. Linear regression can still be useful even when you suspect the true data-generating distribution doesn't conform to the model.

Here, item 2 is just a general consequence of the fact that models need not be a correct representation of the actual real-life process generating the data to be useful. In fact, it is often said that all models are wrong. They can still be useful, though.

To see why item 1 is the case, suppose you have two variables $X$ and $Y$, and you'd like to model $Y$ as normally distributed with mean equal to $a_0 + a_1X + a_2X^2 + a_3\log X$, where the $a_i$s are model parameters you want to choose with least squares. This describes a nonlinear relationship between $X$ and $Y$, but is still a linear model that can be fit with linear regression, because the mean of the dependent variable ($Y$) is modeled as a linear function of three data vectors (namely, $X$, $X^2$, and $\log X$). An example of a truly nonlinear model would be specifying the mean as $a_0 + X^{a_1}$, where you want to choose $a_0$ and $a_1$ with least squares. Here, we have a model parameter ($a_1$) that has a different relationship with a data vector ($X$) than simple multiplication. Hence, you need a model-fitting technique other than ordinary least squares.

Consider a model like $y = \alpha + \exp(\gamma+\beta x) + \epsilon$...

to have the second term linear in $\beta$ and $\gamma$ you'd want to take logs ... but to have it linear in $\alpha$ (and the error term) you wouldn't want to take logs. You can't do both at once.

(Note, however, that given beta, this is linear in $\alpha$ and $\exp(\gamma)$ and given $\alpha$, $\log(E(y)-\alpha)$ is linear in $\beta$ and $\gamma$, though on that last issue see the comments below.)

The final term there raises an important issue with linearization -- while many nonlinear models can be "linearized" by transformation, because of the presence of the error term (noise term), if you fit them the same way (e.g. if you start with an assumption of additive homoskedastic error and then make that assumption after you linearize) they're no longer the same model.

The problem comes about because people leave out half the model when they write it.

That is, people tend to write something like $y= f(a+bx)$ (e.g. $y = \frac{1}{a+bx}$ or $y = Ae^bx$) and then say "just transform both sides" (e.f. "just invert both sides" or "just take logs of both sides") and that looks fine if you write the model like that ... but that's actually not the model for the data!

The $y_i$ values are your responses, and the $x_i$ values are your predictors, and it's not the case that $y_i = \frac{1}{a+bx_i}$ (or whatever other $f$ you might have).

You need the model to also account for $y_i - E(y_i|x_i)$.

If you propose an additive error for which least squares would make sense in the original problem, you don't have additive error on transforming it. Or if it makes sense to have additive error after your transformed, it would not have before you transformed. (There's also the related issue of back-transforming and expected values that is often neglected.)

The exception to this is where the noise is no small that you're essentially getting observations with no error at all (in which case you don't really need regression, just linearize and draw the line in).