Abstract example. Let's say, I want to compare MLE (Maximum Likelihood Estimation) and LSE (Least Squares Estimation) performance on the small samples for simple linear regression.

My idea: to generate two normally distributed variables x and y with very large number of observations. As far as I know, MLE and LSE are asymptotically equal, so they should return almost equal parameter estimations on the whole dataset.

Then I will take n random samples of size m (e.g. 1000 samples for each m from 10 to 1000) from the "population" that was generated on the previous step, build two (MLE-based and LSE-based) regression models on each sample and compare "real" parameter values and parameters from each model.

If for one of the methods (MLE or LSE) the mean of estimated parameter on the small samples will not be equal to the "real" parameter value, can I say, that method is giving biased estimations on the small samples?

If for one of the methods the variance of estimated parameter will be greater than for the other, can I say that the method gives less stable estimations?

How can I calculate the significance of this differences? Is this design valid at all? Maybe there are better alternatives?


1 Answer 1


Use a parametric bootstrap procedure. Explained in detail in Chapter 6 of Efron and Tibshirani (1993):

  1. Denote the parameters generated from the MLE procedures as B1, ..., B1000
  2. Denote the parameters generated from the LSE procedures as C1, ..., C1000
  3. Order B1,...,B1000 from lowest to highest, denote these as b1,...,b1000
  4. Order C1,...,C1000 from lowest to highest, denote these as c1,...,c1000
  5. Generate a 90% (or whatever) parametric bootstrap CI for the MLE estimate = [b51, b950]
  6. Same for LSE estimate = [c51, c950]
  7. If the two CI's overlap, you can't reject the null hypothesis that the two estimation procedures produce different results for samples of size m. If they don't overlap, you can.
  8. If [b51, b950] doesn't contain the true parameter value, you can conclude, at the 10% level, that the MLE procedure is biased for sample size of m. If it does, can't reject null of unbiasedness. For LSE do the same thing but with [c51, c950].
  9. For tests concerning the sample variance, I think you can apply the same steps, just use var(B1), ..., var(B1000) in Step 1; var(C1), ..., var(C1000) in Step 2 and repeat steps 3-8. But there may be a better method for the variance.
  • $\begingroup$ Not related to Brad Efron I assume. I know a little about the bootstrap too. I am having a hard time understanding the OPs problem. First he say that he wants to compare the least squares estimate to maximum likelihood estimates in simple linear regression when the sample size is small. He doesn't explain what he is assuming with his simulation or what the actual sample size might be? Suddenly two normal distributions appear with large sample sizes. $\endgroup$ Commented Dec 16, 2016 at 15:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.