Why does the least square estimator not depend on the distribution of the error term $\epsilon$ nor need specification of a likelihood?

Question

Suppose that we are interested in non-linear regression where we have the pairs $(X_i, Z_i)_{i=1}^n$ observed and consider the regression of $Z$ on $X$:

$$ \forall i \in \{1, \ldots, n\} \ \ Z_i = f_{\theta}(X_i) + \epsilon_i $$

for generic functions $f_\theta, \theta \in \mathcal{H}$, where $\mathcal{H}$ is the parameter space and noise terms $(\epsilon_i)_{i=1}^{n}$. The least squares estimator is defined as:

$$ argmin_{\theta\in \mathcal{H}}\sum_{i=1}^{n}(Z_i-f_\theta(X_i))^2 $$

I was told that in order to consider this estimator, we don't have to specify the distribution of $\epsilon_i$ nor do we have to specify a likelihood.

Can someone tell me why this is the case? Thanks.

Answer 1

Well, the least squares estimator is an estimator defined as the one that minimizes the squares of the residuals. That is just its definition. We can calculate it based on this definition and then start considering any properties it might have in your parameter space.

So we can define it, estimate it and investigate any properties it might have... all without specifying the error distribution or any likelihood. As such, you might call it "almost non-statistical".

Alternatively, we could define, estimate and investigate a least absolute error estimator. Or a least $\ell_p$ error estimator for yet any other $p\neq 1,2$. No problem whatsoever.

However, you will likely not get very far in discussing the properties of these estimators if you don't connect it in some way to the error distribution (and therefore set up a likelihood). In OLS, properties like least squares being BLUE depend crucially on the error distribution (or their large-sample asymptotics).

Bottom line: nothing keeps you from defining an estimator in a certain way, but you will likely not be able to derive many interesting properties unless you also assume error distributions and other properties of your model.