Suppose that we are interested in non-linear regression where we have the pairs $(X_i, Z_i)_{i=1}^n$ observed and consider the regression of $Z$ on $X$:

$$ \forall i \in \{1, \ldots, n\} \ \ Z_i = f_{\theta}(X_i) + \epsilon_i $$

for generic functions $f_\theta, \theta \in \mathcal{H}$, where $\mathcal{H}$ is the parameter space and noise terms $(\epsilon_i)_{i=1}^{n}$. The least squares estimator is defined as:

$$ argmin_{\theta\in \mathcal{H}}\sum_{i=1}^{n}(Z_i-f_\theta(X_i))^2 $$

I was told that in order to consider this estimator, we don't have to specify the distribution of $\epsilon_i$ nor do we have to specify a likelihood.

Can someone tell me why this is the case? Thanks.

  • 1
    $\begingroup$ I deleted my answer because I thought you were referring to linear regression. But I guess you are allowing $f_\theta$ to be a nonlinear function of $\theta$. $\endgroup$ May 2, 2017 at 2:45
  • $\begingroup$ What is the space of nonlinear functions that you are considering? What is $\mathcal{H}$ ? $\endgroup$ May 2, 2017 at 3:00
  • $\begingroup$ I found this mention in a textbook, which only said that $f_\theta$ was a set of generic functions. Do they have to be classified according to a space? $\endgroup$
    – user321627
    May 2, 2017 at 3:02
  • $\begingroup$ You said that $\theta$ belongs to $\mathcal{H}$. $\endgroup$ May 2, 2017 at 3:07
  • $\begingroup$ @MichaelChernick $\mathcal{H}$ refers to the parameter space here, thanks. $\endgroup$
    – user321627
    May 2, 2017 at 3:08

1 Answer 1


Well, the least squares estimator is an estimator defined as the one that minimizes the squares of the residuals. That is just its definition. We can calculate it based on this definition and then start considering any properties it might have in your parameter space.

So we can define it, estimate it and investigate any properties it might have... all without specifying the error distribution or any likelihood. As such, you might call it "almost non-statistical".

Alternatively, we could define, estimate and investigate a least absolute error estimator. Or a least $\ell_p$ error estimator for yet any other $p\neq 1,2$. No problem whatsoever.

However, you will likely not get very far in discussing the properties of these estimators if you don't connect it in some way to the error distribution (and therefore set up a likelihood). In OLS, properties like least squares being BLUE depend crucially on the error distribution (or their large-sample asymptotics).

Bottom line: nothing keeps you from defining an estimator in a certain way, but you will likely not be able to derive many interesting properties unless you also assume error distributions and other properties of your model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.