1
$\begingroup$

In statistical parameter estimation where there is a deterministic and stochastic component to the observation-generating model, do least squares and maximum likelihood estimators always exist? Solving inverse problems more generally begins with the question of existence but in this case, these estimators are defined by objective functions to be minimized or maximized - would there not always be a solution that is "closest" to the data even if the model is not a good descriptor of the data. So can we say that the solution always exists? I am not certain why here it says that for existence of the MLE, the likelihood function should be continuous and the domain of the parameter compact (unless the estimator must be expressed analytically?).

Furthermore, since least squares estimators don't make any statements about the statistical distribution of errors/residuals (unless you are trying to make a statement about its bias or variance), solutions would always exist for even non-linear least squares problems. Is this correct?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The likelihood being continuous over a compact parameter space is a sufficient but not necessary condition for the MLE to exist. This means that whenever you have a likelihood that is continuous and a parameter space that is compact, you know for sure that a MLE exists - but the MLE can also exist in other cases. For instance, if the likelihood is twice differentiable, concave and its gradient is zero somewhere, the MLE will also exist.

For example, imagine you have a sample of Bernoulli experiments (e.g. a few coin tosses, where we code heads as 1 and tails as 0). If we consider our parameter (the probability of getting heads) as being any $0\leq\theta\leq1$ and we got zero heads, we get MLE $\hat\theta=0$. Had we considered the parameter being $0<\theta<1$, the MLE would not exist: the closer $\hat\theta$ gets to zero, the larger the likelihood would get, but zero is not an option in the parameter space.

That being said, all likelihoods used in practice have a well defined MLE, even if they cannot be expressed analytically and have to be found numerically. (Perhaps there is some obscure exception for this that I'm not aware of)

About least squares: in all cases I can think of, least squares methods will reveal a convex optimization problem for which a solution always exists. Again, maybe there is some obscure exception, but I've never found one in practice.

Hope I was helpful!

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ "For instance, if the likelihood is twice differentiable, concave and its gradient is zero somewhere, the MLE will also exist."---For existence, differentiability is not needed. Concavity and non-monotonicity suffices (for convex optimization problems in general.) $\endgroup$
    – Michael
    Commented Oct 6, 2020 at 20:39
  • $\begingroup$ When the variance of your $x$ variable(s) are zero then you have basically a single point and then OLS estimator does not exist, but also in practice this is not likely. I also missed the part about the sufficient condition part, makes sense. $\endgroup$
    – hatmatrix
    Commented Oct 7, 2020 at 2:59
  • 1
    $\begingroup$ @hatmatrix one could argue that in that case the OLS estimator exists but is not unique, but I get your point $\endgroup$
    – PedroSebe
    Commented Oct 7, 2020 at 12:13
  • $\begingroup$ @PedroSebe in concept I believe that's true, for OLS of a single variable for instance, the slope is calculated as the $Cov[x,y]/Var[x]$ and would be undefined if $Var[x] =0$? $\endgroup$
    – hatmatrix
    Commented Oct 9, 2020 at 10:33
  • $\begingroup$ @hatmatrix well, it's kind of a technicality, but: if $Var(x)=0$, then any $\beta_1$ can achieve the minimum MSE if you set $\beta_0=\bar y-\beta_1\bar x$. This indetermination is the reasom why the usual formula gives 0/0. So, the minimum does still exist but is not unique, which makes the estimator ill-defined and pretty much useless. $\endgroup$
    – PedroSebe
    Commented Oct 9, 2020 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.