Nonlinear least squares vs linear least squares

Question

I would like to fit a model to data that has the form,

$$f(x)=e^{a+bx+cx^2...}$$

The data is Gaussian distributed, and I was going to use a nonlinear least squares method (probably something in scipy). Each $y(x)$ has and associated $\sigma$ so I would like to do a weighted nonlinear least-squares.

The standard method seems to be to take the logarithm of the data and fit a polynomial to it using linear least-squares. However, taking the logarithm of the data would do something to the uncertainties (I haven't even really looked into how the uncertainties would transform under logarithm). Additionally, maybe I want to play around with my model a bit at fit something more of the form

$$f(x)=(a+bx)e^{cx^2}$$

or whatever.

Thus I would prefer to keep everything as general as possible and just fit the model to the data directly.

Linear least-squares involves minimizing the $L^2$ norm of $\mathbf{b}-A\mathbf{x}$, and, if you have a poorly conditioned model ($A$), then you can get garbage results from numerical methods. I assume that matrix conditioning is also important in nonlinear least squares?

I have a range questions. Is there anything inherently poorly conditioned about exponential models in general? Are nonlinear least squares methods inherently worse than linear least squares methods? The linear least squares method minimizes the $\chi^2$ statistic which maximizes the likelihood if the data is Gaussian distributed. Does the nonlinear least squares method still give a maximum likelihood model fit if the data is Gaussian?

Answer 1

1. When you say each $x$ has an associated $\sigma$, I assume you actually mean the associated $y$ has an associated $\sigma$. Fitting with $x$-uncertainties is a less common problem.

2. Taking a logarithm just maps relative uncertainties into absolute uncertainties. In other words, if $z = \ln y$, then $(\delta z) = (\delta y)/y$. You can see this by simply differentiating.

3. A minimization problem that is poorly-conditioned when formulated as a linear problem is going to be just as poorly conditioned (and harder to solve) when formulated as a nonlinear problem.

4. Maximum likelihood does not require linearity. It's still a maximum likelihood fit, with all the guaranteed asymptotic goodness of maximum likelihoods, if the model is non-linear.

5. You will likely get slightly different results from fitting the nonlinear, exponential form and fitting the linearized, log-transformed form. That's because the uncertainties, interpreted as normal distributions $N(y,\delta y)$ and $N(z, \delta z)$, don't give exactly the same probability to corresponding values. (You can't have perfectly normally distributed residuals in both the original and the log-transformed variables.) In most circumstances, the researcher won't be certain of perfectly normally distributed residuals in either space, and the difference will be much smaller than the uncertainties anyway.