2
$\begingroup$

This is my data:-

c(3164, 3362, 4435, 3542, 3578, 4529)

I estimated its sample mean and standard deviation via mean & sd functions and got the following results

mu
[1] 3768.333
sigma
[1] 572.859

now I generated the parameter values that maximize the log likelihood assuming the underlying data is generated from a normal distribution using the following code and optim function

op <- function(para){
    loglik <- sum(log(dnorm(x,para[1],para[2])))
    -loglik
}

optim(c(3200,570),op)

OUTPUT:-

$par
[1] 3768.2043  522.9118

$value
[1] 46.0705

$counts
function gradient 
      49       NA 

$convergence
[1] 0

$message
NULL

My question is shouldn't the sample sd 572.859 be the parameter value that maximises the log-likelihood rather than 522.9118, and if not then how can we assume that the sample sd is an unbiased estimator of the population sd?

Improve this question
$\endgroup$
1
  • $\begingroup$ Here the MLE for $\sigma^2$ is biased since it divides by $n$, whereas the unbiased estimator for $\sigma^2$ divides by $n-1$. $\endgroup$
    – utobi
    Sep 1 at 10:26

1 Answer 1

Reset to default
0
$\begingroup$

The maximum likelihood estimator to which you refer (for a Gaussian likelihood) can be derived as $\hat\sigma =\sqrt{ \dfrac{ \sum_{i=1}^N\left( X_i-\bar X\right)^2 }{ N } }$.

The function you’ve called divides by $N-1$. This is because dividing by $N-1$ and not taking the square root results in an unbiased estimator of variance, and then the square root of the unbiased estimator of variance is taken as a fairly natural step to estimate the square root of the variance (though this is a biased estimator of standard deviation by Jensen’s inequality).

That is, the sd function is not calculating according to maximum likelihood estimation.

If you increase the sample size, you should see the two results get closer and closer together.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Does this mean that I can't use MLE to estimate parameter values? And if so how can I estimate it given some data? $\endgroup$
    – Rishav Dhariwal
    Sep 2 at 12:44
  • $\begingroup$ Also then for gradient descent methods when can we trust their results if in some cases as above they can give biased estimates of parameter values? $\endgroup$
    – Rishav Dhariwal
    Sep 2 at 12:46
  • $\begingroup$ @RishavDhariwal There’s nothing inherently wrong with biased estimators. In fact, many popular estimators (e.g., ridge regression) go out of their way to introduce a bit bias in hopes of scoring major improvements in other estimator properties, such as variance. $\endgroup$
    – Dave
    Sep 2 at 15:30
  • $\begingroup$ I understand that but still I would want to know when applying MLE could introduce bias in my estimates and for which ones should i be careful. For example, here I am getting a biased estimate of SD but other distributions have many different parameters like degree of freedom etc is there any test I can perform on my estimates for this? $\endgroup$
    – Rishav Dhariwal
    Sep 3 at 8:09
  • $\begingroup$ @RishavDhariwal “Bias” doesn’t mean the estimator gives you the incorrect value. In fact, you can be basically certain that, no matter how you estimate a parameter, your estimate is incorrect. With that in mind, I do not really understand what you’re asking. Perhaps it would be worth fleshing out your inquiry into a full question to post. $\endgroup$
    – Dave
    Sep 5 at 4:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.