(A physicist self-studying statistics here)

I was previously confused about the meaning of the standard error in a maximum likelihood estimate. Certain stack exchange posts (linked below) have gone some way in clearing this, but I am still unsure about a few details. My confusion stemmed from the fact that :

  • In any given experiment, all of the data are used in calculating the likelihood, and thus we only get a single value for the MLE. I am assuming that the number of trials/data samples exceeds the number of covariates $n>p$.
  • Clearly, for covariates $p\geq 1$, the MLE does not exist if $n<p$. So it is generally not possible to assign an MLE for single data samples (i.e. set of covariate values$\textbf{x}$ and dependent variable value $y$)
  • The variance is a measure of spread. But what is this spread of if we just derive a single MLE?

The answer here on 'What is meant by the standard error of a maximum likelihood estimate' was very relevant. The question was the same as mine, although the answers still talked on 'the asymptotic distribution of the MLE being Gaussian...' with the given variance.

So I just want to make clear if my understanding is correct:

  • The likelihood, and the MLE, is a function of the data also, which includes the number of data samples.
  • So in talking of the variance of an MLE, we are referring to a spread of MLE values computed in different trials, each with the same number n of data samples.

    1. For example, if I have p=5 covariates and obtain n=100 data samples, I will have a 100x6 design matrix $\textbf{X}$ (including intercept) and 100 output values in the vector $\textbf{y}$. I will use this to compute a single MLE.
    2. Say I then repeat this set up 20 times, i.e. 20 trials of 100 data samples each. Then I will compute the sample standard deviation of my 20 estimates of the MLE (sample s.d. because I am not assuming the true value of the parameters).
    3. Since the MLE is gaussian distributed, my calculated variance should remain approx constant as I increase the number of trials (from 20).
    4. However, for any set number of trials, this covariance matrix tends to $\mathcal{I}^{-1}/n$ where $\mathcal{I}$ is the Fisher information matrix. I believe that the Fisher matrix scales as $o(n)$, which would imply that the variance is constant in the asymptotic limit of the sample number? This doesn't seem right, and is inconsistent with an answer in the linked post. Though This Wiki article does refer to an additive Fisher information matrix, proportional to n!

    5. I would also appreciate some clarification on the mathematical distinction- for different models- between increasing the number of samples for a fixed number of trials, and increasing the number of trials for fixed number of samples. Clearly, using m trial data for n samples each, all at once, is equivalent to using nxm samples data for a single trial. But what about combining m trials?

Additional background note: These questions came up when I wanted to run some simulations for the Gaussian and logistic models to test the ideas and theorems of MLEs.

  • $\begingroup$ 1. Be careful about the use of the word "sample", which I think may be confusing you. You seem to be using it the way we use the word observation which might cause you to misinterpret what you're reading (and if not, your usage may mislead some readers of your post). 2. I don't see how the conclusion in your point 3 follows from the premise. How do you get from "is Gaussian distributed" to the conclusion that the distribution doesn't depend on sample size? $\endgroup$ – Glen_b -Reinstate Monica Jul 14 '19 at 0:39
  • $\begingroup$ @Glen_b Thank you for your comment and the useful link. Using correct terminology, I assumed that for a given observation number, there was a corresponding Gaussian distribution from which the MLEs were drawn, with a fixed variance. So the mean of square deviations from the sample mean would remain the same. $\endgroup$ – Meep Jul 14 '19 at 13:49
  • $\begingroup$ I think I follow your intent better now. It may be better to mention your interest in (Gaussian) linear regression and logistic regression up front rather than at the very end of the question; the reference to covariates seems out of place in the absence of a model that would include them in it. If you want the question to be more general you could posit some model that includes covariates to begin with. ... Could you clarify the inconsistency you mention in 4? I don't see the inconsistency. $\endgroup$ – Glen_b -Reinstate Monica Jul 14 '19 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.