Are least squares equivalent to ML normal distribution for any $f$?

I know that it's true if the model is linear and about least squares to non-linear models?

I mean:

$$LS=argmin_{\theta} \sum_{i=1}^{N} (y_i-f(x_i,\beta))^2$$ , where $$x_i\in \Re^n$$

is equivalent to:

$$ML=argmax_{\theta}\prod_{i=1}^{N}\phi(y_i,f(x_i,\beta),\sigma)$$, where $$\phi$$ is normal distribution function and $$y_i\sim N(f(x_i,\beta),\sigma)$$

for any $$f\in \mathcal{F}$$. Note i didnt use multivariate normal function even $$x_i \in \Re^n$$.

• Take the negative logarithm of ML and draw the conclusion.
– whuber
Commented Jul 1, 2021 at 14:22

Least Squares Estimation (LS)

The least squares estimation aims to minimize the sum of squared residuals between the observed values $$y_i$$ and the model predictions $$f(x_i, \beta)$$:

$$LS = \arg\min_{\beta} \sum_{i=1}^{N} (y_i - f(x_i, \beta))^2$$

Here, $$\beta$$ represents the parameters of the model, and $$f(x_i, \beta)$$ is a non-linear function predicting $$y_i$$ based on $$x_i$$ and $$\beta$$.

Maximum Likelihood Estimation (ML)

The maximum likelihood estimation seeks to maximize the likelihood function, which for normally distributed errors can be written as:

$$ML = \arg\max_{\beta} \prod_{i=1}^{N} \phi(y_i, f(x_i, \beta), \sigma)$$

where $$\phi(y_i, f(x_i, \beta), \sigma)$$ is the normal probability density function given by:

$$\phi(y_i, f(x_i, \beta), \sigma) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{(y_i - f(x_i, \beta))^2}{2\sigma^2}\right)$$

Here, $$y_i \sim N(f(x_i, \beta), \sigma)$$ indicates that the $$y_i$$ are normally distributed around the mean $$f(x_i, \beta)$$ with a standard deviation $$\sigma$$.

To show the equivalence, we need to connect the least squares estimation to the log-likelihood function for normally distributed errors.

Negative Log-Likelihood

Taking the negative logarithm of the likelihood function simplifies the product of exponentials into a sum, which is often easier to maximize. The log-likelihood function $$L(\beta)$$ for the normal distribution is:

$$L(\beta) = \sum_{i=1}^{N} \log \left( \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left( -\frac{(y_i - f(x_i, \beta))^2}{2\sigma^2} \right) \right)$$

This can be simplified to:

$$L(\beta) = \sum_{i=1}^{N} \left( -\frac{1}{2} \log(2 \pi \sigma^2) - \frac{(y_i - f(x_i, \beta))^2}{2\sigma^2} \right)$$

The first term, $$-\frac{1}{2} \log(2 \pi \sigma^2)$$, is a constant and does not depend on $$\beta$$. Therefore, maximising $$L(\beta)$$ is equivalent to minimising the second term:

$$\arg\max_{\beta} L(\beta) = \arg\min_{\beta} \sum_{i=1}^{N} \frac{(y_i - f(x_i, \beta))^2}{2\sigma^2}$$

Since $$\frac{1}{2\sigma^2}$$ is a positive constant, minimising $$\sum_{i=1}^{N} (y_i - f(x_i, \beta))^2$$ will also minimize $$\sum_{i=1}^{N} \frac{(y_i - f(x_i, \beta))^2}{2\sigma^2}$$.

Therefore, we see that:

$$\arg\min_{\beta} \sum_{i=1}^{N} (y_i - f(x_i, \beta))^2$$

is indeed equivalent to:

$$\arg\max_{\beta} \sum_{i=1}^{N} \log \left( \phi(y_i, f(x_i, \beta), \sigma) \right)$$

Hence, taking the negative logarithm of the likelihood function transforms the maximisation of the likelihood into the minimisation of the sum of squared residuals, establishing the equivalence between least squares estimation and maximum likelihood estimation under the assumption of normally distributed errors.

Conclusion

The equivalence between least squares (LS) and maximum likelihood (ML) for any non-linear model $$f$$ under normally distributed errors can be understood by transforming the product of the normal distribution functions into a sum through the logarithm. This transformation leads to a form that reveals the least squares minimisation problem, thus showing their equivalence.