Inspired by the question regarding Bessel's correction, I wonder whether there is a general rule regarding applicability of maximum likelihood for parameter estimation.

My guess is that the parameters estimated by maximising the likelihood in linear regression are optimal because they are independent. It seems intuitive to me, although not 'obvious', that the same holds for all generalised linear models.

On the other hand, the estimate of population variance depends on the estimate of the mean and this seems somehow to affect maximum likelihood (I'm aware of the math showing the need for correction, but it's still not intuitively clear to me).

So, is it correct to say that the independence of the parameter estimates is a requirement for maximum likelihood? If not, what conditions need to be satisfied?

  • $\begingroup$ I’m not sure what your question is. A maximum likelihood estimate is optimal in the sense that it maximizes the likelihood function given a sample of data. If you get a sample $X$ and from it you somehow compute an estimate of the parameter $\theta$, denoted $\hat \theta$, and then plug $\hat \theta$ into the likelihood function with $X$, if the likelihood function is maximized, then $\hat \theta$ is a maximum likelihood estimate. $\endgroup$
    – mhdadk
    Sep 19, 2022 at 11:30
  • 2
    $\begingroup$ In the frequentist world in which maximim likelihood estimation is used, the parameters don't have distributions, so it is not meaningful to refer to them as dependent or independent of each other. So I gather you are referring to independence of parameter estimates. There is no need for that either, for MLE to be optimal. The main problem with MLE is that it is equivalent to Bayesian posterior mode where all values of all parameters are equally likely before knowing hte data. This is a very unreasonable assumption. $\endgroup$ Sep 19, 2022 at 11:33
  • 1
    $\begingroup$ "Maximum-likelihood estimators have no optimum properties for finite samples, in the sense that (when evaluated on finite samples) other estimators may have greater concentration around the true parameter-value.[16] However, like other estimation methods, maximum likelihood estimation possesses a number of attractive limiting properties: As the sample size increases to infinity, sequences of maximum likelihood estimators have these properties... Wikipedia Maximum Likelihood $\endgroup$
    – seanv507
    Sep 19, 2022 at 13:44
  • 1
    $\begingroup$ Maybe relevant: stats.stackexchange.com/questions/92097/… $\endgroup$ Sep 19, 2022 at 14:07
  • 1
    $\begingroup$ @IgorF. we use MLE all the time when two of the parameter estimates are correlated (e.g., two regression coefficients for two collinear Xs) so don't get hung up on that. $\endgroup$ Sep 20, 2022 at 12:12


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.