# Why does the least square estimator not depend on the distribution of the error term $\epsilon$ nor need specification of a likelihood?

Suppose that we are interested in non-linear regression where we have the pairs $(X_i, Z_i)_{i=1}^n$ observed and consider the regression of $Z$ on $X$:

$$\forall i \in \{1, \ldots, n\} \ \ Z_i = f_{\theta}(X_i) + \epsilon_i$$

for generic functions $f_\theta, \theta \in \mathcal{H}$, where $\mathcal{H}$ is the parameter space and noise terms $(\epsilon_i)_{i=1}^{n}$. The least squares estimator is defined as:

$$argmin_{\theta\in \mathcal{H}}\sum_{i=1}^{n}(Z_i-f_\theta(X_i))^2$$

I was told that in order to consider this estimator, we don't have to specify the distribution of $\epsilon_i$ nor do we have to specify a likelihood.

Can someone tell me why this is the case? Thanks.

• I deleted my answer because I thought you were referring to linear regression. But I guess you are allowing $f_\theta$ to be a nonlinear function of $\theta$. May 2, 2017 at 2:45
• What is the space of nonlinear functions that you are considering? What is $\mathcal{H}$ ? May 2, 2017 at 3:00
• I found this mention in a textbook, which only said that $f_\theta$ was a set of generic functions. Do they have to be classified according to a space? May 2, 2017 at 3:02
• You said that $\theta$ belongs to $\mathcal{H}$. May 2, 2017 at 3:07
• @MichaelChernick $\mathcal{H}$ refers to the parameter space here, thanks. May 2, 2017 at 3:08

Alternatively, we could define, estimate and investigate a least absolute error estimator. Or a least $\ell_p$ error estimator for yet any other $p\neq 1,2$. No problem whatsoever.