# In nonlinear regression, when is MLE equivalent to least squares regression?

I recently received this one line question in a job interview and was a little stumped by it.

In nonlinear regression, when is Maximum Likelihood Estimation equivalent to least squares?

• It seems like [self-study] question, so what is your response to this? What kind of hints would help you?
– Tim
Commented Mar 1, 2022 at 9:21

By definition, the least squares estimator minimises the sum of the squared distances between the actual and predicted responses. With a set of simple steps, you can show that this estimator is equivalent to the solution of a certain maximisation problem. If we let $$f$$ denote the nonlinear regression function and let $$\boldsymbol{\beta}$$ denote the parameter of this function (and let $$\sigma>0$$ be an arbitrary scaling parameter), we then have:

\begin{align} \hat{\boldsymbol{\beta}}_\text{OLS}(\mathbf{y}, \mathbf{x}) &\equiv \underset{\boldsymbol{\beta}}{\text{arg min}} \sum_{i=1}^n (y_i - f(\mathbf{x}_i, \boldsymbol{\beta}))^2 \\[6pt] &= \underset{\boldsymbol{\beta}}{\text{arg max}} \bigg( - \sum_{i=1}^n (y_i - f(\mathbf{x}_i, \boldsymbol{\beta}))^2 \bigg) \\[6pt] &= \underset{\boldsymbol{\beta}}{\text{arg max}} \bigg( - \frac{1}{2 \sigma^2} \sum_{i=1}^n (y_i - f(\mathbf{x}_i, \boldsymbol{\beta}))^2 \bigg) \\[6pt] &= \underset{\boldsymbol{\beta}}{\text{arg max}} \ \exp \bigg( - \frac{1}{2 \sigma^2} \sum_{i=1}^n (y_i - f(\mathbf{x}_i, \boldsymbol{\beta}))^2 \bigg) \\[6pt] &= \underset{\boldsymbol{\beta}}{\text{arg max}} \ \prod_{i=1}^n \exp \bigg( - \frac{1}{2 \sigma^2} (y_i - f(\mathbf{x}_i, \boldsymbol{\beta}))^2 \bigg) \\[6pt] &= \underset{\boldsymbol{\beta}}{\text{arg max}} \ \prod_{i=1}^n \text{N} (y_i | f(\mathbf{x}_i, \boldsymbol{\beta}), \sigma^2). \\[6pt] \end{align}

(These steps use the fact that the $$\text{arg min}$$ and $$\text{arg max}$$ are invariant/anti-variant to strictly monotonic transformations. Look through the steps to ensure you understand why the minimising/maximising point is preserved under the steps.) The latter estimator is an MLE for a certain nonlinear regression model form --- can you see what model form this is?

Update: Per the suggestion from Dave in comments below, now that this question is a year old we can give a full solution. From the above equation we see that the MLE matches the least squares estimator when the regression model uses IID normal (Gaussian) error terms.

• What is N() meant to imply in the last step?... oh, the distribution type with standard deviation, sigma. I don't really get how one would interpret the second last line to be equivalent to the last line though. does "|" mean something different in this context from the standard x given y normally used in probability? Commented Mar 1, 2022 at 18:27
• @Wesley: The value $y_i$ coming before the $|$ part is the argument of the density function (i.e., the outcome value whose density we are interested in). This is standard notation when we use density functions.
– Ben
Commented Mar 1, 2022 at 20:33
• Just isn't my field, but I can accept it. The second last line is mathematically clear to a pleb like me, the last line, jargon. Commented Mar 1, 2022 at 20:49
• @Wesley: The notation on the last line is for the normal density function. If you have a look at the form of this density you will see that it is the same as the line above, except for a positive scaling constant. (Also, for anyone outside of maths/stats, everything in the question and answer is jargon.)
– Ben
Commented Mar 1, 2022 at 20:53
• So to summarise, my understanding: 1) you can rearrange (via monotonicity, negation, and exponent addition) this square loss minimisation to maximising the product of probabilities of the example labels (the probabilities are assigned by your model, given its learned parameters $\beta$ and an arbitrary $\sigma$) 2) in short, the two are equivalent when your model's errors are normal? Commented Mar 3, 2022 at 15:59