How to compare effectiveness of methods of parameter estimation on the small samples?

Question

Abstract example. Let's say, I want to compare MLE (Maximum Likelihood Estimation) and LSE (Least Squares Estimation) performance on the small samples for simple linear regression.

My idea: to generate two normally distributed variables x and y with very large number of observations. As far as I know, MLE and LSE are asymptotically equal, so they should return almost equal parameter estimations on the whole dataset.

Then I will take n random samples of size m (e.g. 1000 samples for each m from 10 to 1000) from the "population" that was generated on the previous step, build two (MLE-based and LSE-based) regression models on each sample and compare "real" parameter values and parameters from each model.

If for one of the methods (MLE or LSE) the mean of estimated parameter on the small samples will not be equal to the "real" parameter value, can I say, that method is giving biased estimations on the small samples?

If for one of the methods the variance of estimated parameter will be greater than for the other, can I say that the method gives less stable estimations?

How can I calculate the significance of this differences? Is this design valid at all? Maybe there are better alternatives?

Answer 1

Use a parametric bootstrap procedure. Explained in detail in Chapter 6 of Efron and Tibshirani (1993):

1. Denote the parameters generated from the MLE procedures as B1, ..., B1000
2. Denote the parameters generated from the LSE procedures as C1, ..., C1000
3. Order B1,...,B1000 from lowest to highest, denote these as b1,...,b1000
4. Order C1,...,C1000 from lowest to highest, denote these as c1,...,c1000
5. Generate a 90% (or whatever) parametric bootstrap CI for the MLE estimate = [b51, b950]
6. Same for LSE estimate = [c51, c950]
7. If the two CI's overlap, you can't reject the null hypothesis that the two estimation procedures produce different results for samples of size m. If they don't overlap, you can.
8. If [b51, b950] doesn't contain the true parameter value, you can conclude, at the 10% level, that the MLE procedure is biased for sample size of m. If it does, can't reject null of unbiasedness. For LSE do the same thing but with [c51, c950].
9. For tests concerning the sample variance, I think you can apply the same steps, just use var(B1), ..., var(B1000) in Step 1; var(C1), ..., var(C1000) in Step 2 and repeat steps 3-8. But there may be a better method for the variance.