# Nonlinear least squares vs linear least squares

I would like to fit a model to data that has the form,

$$f(x)=e^{a+bx+cx^2...}$$

The data is Gaussian distributed, and I was going to use a nonlinear least squares method (probably something in scipy). Each $y(x)$ has and associated $\sigma$ so I would like to do a weighted nonlinear least-squares.

The standard method seems to be to take the logarithm of the data and fit a polynomial to it using linear least-squares. However, taking the logarithm of the data would do something to the uncertainties (I haven't even really looked into how the uncertainties would transform under logarithm). Additionally, maybe I want to play around with my model a bit at fit something more of the form

$$f(x)=(a+bx)e^{cx^2}$$

or whatever.

Thus I would prefer to keep everything as general as possible and just fit the model to the data directly.

Linear least-squares involves minimizing the $L^2$ norm of $\mathbf{b}-A\mathbf{x}$, and, if you have a poorly conditioned model ($A$), then you can get garbage results from numerical methods. I assume that matrix conditioning is also important in nonlinear least squares?

I have a range questions. Is there anything inherently poorly conditioned about exponential models in general? Are nonlinear least squares methods inherently worse than linear least squares methods? The linear least squares method minimizes the $\chi^2$ statistic which maximizes the likelihood if the data is Gaussian distributed. Does the nonlinear least squares method still give a maximum likelihood model fit if the data is Gaussian?

• For nonlinear least squares, maximum likelihood would mean the conditional data distribution $y|x$ is Gaussian, not the data distribution $y$ itself. These are not the same thing (e.g. see here). Commented May 2, 2017 at 20:24
• To verify that the data was Gaussian distributed I performed an Anderson-Darling test for normality on a number of "slices" of data (at particular $x$ values). I didn't really think about it, but I guess this means that the conditional data distribution is Gaussian. I did this because I had a hunch that the data was not Gaussian distributed for some values of $x$ (which was true, I am not fitting these sections).
– Alex
Commented May 2, 2017 at 20:48
• As a general principle, avoid posting a shopping list of different questions. Try to stick to one main issue at a time. Commented May 3, 2017 at 4:00

1. When you say each $x$ has an associated $\sigma$, I assume you actually mean the associated $y$ has an associated $\sigma$. Fitting with $x$-uncertainties is a less common problem.

2. Taking a logarithm just maps relative uncertainties into absolute uncertainties. In other words, if $z = \ln y$, then $(\delta z) = (\delta y)/y$. You can see this by simply differentiating.

3. A minimization problem that is poorly-conditioned when formulated as a linear problem is going to be just as poorly conditioned (and harder to solve) when formulated as a nonlinear problem.

4. Maximum likelihood does not require linearity. It's still a maximum likelihood fit, with all the guaranteed asymptotic goodness of maximum likelihoods, if the model is non-linear.

5. You will likely get slightly different results from fitting the nonlinear, exponential form and fitting the linearized, log-transformed form. That's because the uncertainties, interpreted as normal distributions $N(y,\delta y)$ and $N(z, \delta z)$, don't give exactly the same probability to corresponding values. (You can't have perfectly normally distributed residuals in both the original and the log-transformed variables.) In most circumstances, the researcher won't be certain of perfectly normally distributed residuals in either space, and the difference will be much smaller than the uncertainties anyway.

• You're right, I will edit my post to be uncertainties in $y$. The data is most certainly not Gaussian distributed in log space. I forgot to write this, but this was another motivation for me to do a non-linear least squares. In this situation ("data" from a simulation), I have enough $y$ values for each $x$ that I can actually test the distribution for normality. I am using this to determine what range of data I will actually fit because the distribution changes with $x$.
– Alex
Commented May 2, 2017 at 22:05