Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

It was my understanding that MLE estimates were asymptotically unbiased. The following simulation therefore confuses me. Can anyone help me with my understanding here?

I estimate the parameters of 50 random Weibulls that I've simulated with shape = scale = 1 (using the Wikipedia/R parameterisation)

After lots of simulations, we should* get a distribution of parameter estimates, the median of which equals the simulated values.

The result is that the median of the estimates are significantly different from the simulated parameters:

         shape     scale
est   1.018636 0.9954484
2.5%  1.015124 0.9919716
97.5% 1.021469 0.9984734

I expect I may be wrong at the "should*" above. If so, what is correct?


The R code is below:

I am simulating using rweibull() and estimating parameters using mle() in the core stats4 package and also fitdist() in the fitdistrplus. I am using the correct estimates as the starting parameters, so there shouldn't be any optimisation issues here. (?)

# fitdistrplus::fitdist() fitting
res <- replicate(1e4,
                 fitdist(
                   rweibull(50,shape=1,scale=1),
                   distr="weibull",
                   start=list(shape=1,scale=1)
                 )$est
)

# stats4::mle() fitting
res <- replicate(1e4,{
  y <- rweibull(50,shape=1,scale=1)
  nll <- function(shape,scale) -sum(dweibull(y,shape,scale,log=TRUE))
  res <- stats4::mle(
    nll,
    start=list(shape=1,scale=1)
  )@coef
})

# calculate median of the simulated estimates and its bootstrapped confidence interval
apply(res,1,function(x){
  c(est=median(x),
    quantile(
      replicate(1e3,
                median(sample(x,replace=TRUE))
                ),
      c(0.025,0.975)
      )
    )
})
share|improve this question
add comment

2 Answers 2

up vote 7 down vote accepted

Increase the sample size from fifty & the averages of your parameter estimates over many simulations should get closer still to the true ones—that's what asymptotically unbiased means.

[As @Joe's pointed out, bias (at least "bias" without further qualification—there is such a thing as median-unbiasedness) is the expected difference between the true value & its estimate, so you should be looking at the means rather than the medians in your simulations, if it's this you want to assess.]

share|improve this answer
    
So you're saying that even if I had infinite simulations, but still a finite sample size, it would always be biased? I was originally interpreting asymptotically as meaning "If I simulate enough, it will eventually match the original parameters" –  sqrt May 2 at 13:30
    
That's right. In statistics "asymptotically" means "as the sample size goes to infinity" unless otherwise qualified. You might also be interested in looking at the variance of your estimates, & hence the overall mean-square error. –  Scortchi May 2 at 13:33
    
OK, so are there ways of correcting this consistent bias for the finite sample size case, or is there a general statistical topic name for this? Thanks –  sqrt May 2 at 14:06
    
What makes you think the median of the shapes will be equal to the shape? That doesn't mean that the method is biased for finite samples... What's the sampling distribution of the shape parameter? –  Joe May 2 at 14:25
    
For the general ideas read the chapter on point estimation in a theoretical statistics book. Unbiased estimates sound good but ...(1) They may have higher variance; enough to make the overall MSE higher. (2) Unlike MLEs, they're not in general invariant to reparameterization. So you can get a unbiased estimator of the scale parameter for your Weibull distribution, but its reciprocal won't be an unbiased estimator of the rate. (3) If you choose an estimate that isn't the MLE, your observations were less probable (roughly speaking) if that estimate were right than if the MLE were. –  Scortchi May 2 at 14:28
show 4 more comments

Wikipedia has a section on finite-sample bias correction for MLEs.

http://en.wikipedia.org/wiki/Maximum_likelihood#Higher-order_properties

share|improve this answer

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.