13
$\begingroup$

I recall from my graduate school days that the Gauss-Markov (GM) theorem states that the Best Linear Unbiased Estimator (BLUE) in a linear regression is $\vec{\beta}=(X^TX)^{-1}X^T\vec{y}$. An amazing aspect of the proof is that you do not need distributional assumptions to prove the claim.

I've been studying up on GLMs (and using them) now for several years and you can definitely draw various analogies between GLMs and the linear model.

I've been trying to understand how the GM theorem-or a generalization of it-fits into the GLM theory. It seems like the only place the distribution assumption is used-and even then only sometimes used-is for the link choice. Notably hypothesis testing does not use the distributional assumption instead usually using a Gaussian, or large sample approximation.

My question is if for estimation purposes there is an analogous result to the GM theorem stating BLUE? This may require a generalized definition of linearity but should still be stateable and provable. If anyone is aware of this result a reference would be appreciated.

$\endgroup$
10
  • 2
    $\begingroup$ It's worth noting that MLE estimates of GLM parameters are generally biased: stats.stackexchange.com/questions/60723/… $\endgroup$ Commented Sep 19, 2017 at 4:59
  • 2
    $\begingroup$ @Matthew Drury -> For finite samples the GLM estimates are biased but they are asymptotically unbiased. I guess the equivalent statement to BLUE in GLM world would be an asymptotic statement, good point. $\endgroup$ Commented Sep 19, 2017 at 13:14
  • 2
    $\begingroup$ I wish I knew the answer, I actually got here because I had the same question. I could probably come up with an answer but I'd rather just have the answer. $\endgroup$
    – Digio
    Commented Feb 14, 2019 at 19:13
  • 1
    $\begingroup$ @Digio, FWIW I think the answer is it doesn't really matter other than some conventions but no mathematical theory. For example, a gamma distribution conventionally uses a log-link but that is not the canonical link. As I indicated in the question, the hypothesis tests do not really rely on distribution b/c they are asymptotic and normal cutoff values. Kind of difficult to prove that this doesn't matter though. $\endgroup$ Commented Feb 14, 2019 at 21:28
  • 1
    $\begingroup$ +1 Interesting question! I would think the answer is in the negative. In GLMs other than linear regression, the idea of a "linear" estimator does not make sense to me. If we drop the linearity condition from the Gauss-Markov theorem, depending on what error distribution there is, the usual OLS solution might not be the BUE (best, possibly nonlinear, unbiased estimator). $\endgroup$
    – Dave
    Commented Jan 14, 2022 at 20:18

1 Answer 1

1
$\begingroup$

Since it’s been five years, I will post some speculation, even though I lack a proof.

I would think the answer is in the negative. In GLMs other than linear regression, the idea of a "linear" estimator does not make sense to me. If we drop the linearity condition from the Gauss-Markov theorem, depending on what error distribution there is, the usual OLS solution might not be the BUE (best, possibly nonlinear, unbiased estimator).

Further, the typical maximum likelihood estimation methods in GLMs (I’m thinking of logistic regression) can give biased estimates. If we relax our assumption just to require consistency, then we know maximum likelihood estimation to have nice properties about efficiency among consistent estimators, and then this is almost a tautology. “Is my GLM maximum likelihood estimator the most efficient consistent estimator?” YES! That’s part of the deal with maximum likelihood estimation and why it is a popular approach.

$\endgroup$
4
  • $\begingroup$ How is GLM the most efficient consistent estimator? Wouldn't some constrained estimator (shrinking, ridge, etc.), which is also consistent, do better? $\endgroup$ Commented Sep 5, 2022 at 13:59
  • $\begingroup$ @SextusEmpiricus Maybe there should be some comment in there about “asymptotically” to be fully formal, but I am conceding that this is not one of my more formal answers, which is why I bountied the question last year. $\endgroup$
    – Dave
    Commented Jan 21, 2023 at 7:20
  • $\begingroup$ It looks like I never did bounty this. I might consider doing so, because I do want an answer that is more thorough than mine. $\endgroup$
    – Dave
    Commented Jan 22, 2023 at 21:05
  • 1
    $\begingroup$ In the limit of infinite sample size, the GLM estimate (which is the maximum likelihood estimate) approaches the Cramér-Rao bound. Possibly something like that is an answer but the question becomes a bit broad in the potential interpretations. Something that holds me back in trying to formulate an answer is that GLM is neither a linear estimator nor an unbiased estimator. $\endgroup$ Commented Jan 22, 2023 at 21:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.