I am an MBA Student taking courses in Statistics.

We are learning about different ways to estimate the parameters (i.e. coefficients) of a Regression Model. Our professor indicated that there are two main ways of doing this:

  • Ordinary Least Squares (OLS): In OLS, basically a "line of best fit" is fitted to the data and the regression parameters/coefficients are those from this "line of best fit". As we can see, OLS (i.e. the "line of best fit") requires very few statistical assumptions in the dataset - for example, OLS DOES NOT require the residuals to be Normally Distributed.

  • Maximum Likelihood Estimation (MLE): In MLE, we try to find out the "most likely" set of parameters. For example, suppose we have 100 giraffes - if we ASSUME that the true underlying distribution for the heights of giraffes have a Normal Distribution, then we want to find out the "most likely" (i.e. highest probability) values of the "mean" and the "variance" (i.e. "parameters") for this Normal Distribution that explain the heights of giraffes. In the case of regression, MLE tries to find out the "regression model parameters which likely have the highest probability" out of all regression models to explain this data. Our prof told us that we can imagine a regression model as a Normal Distribution with "mean = beta-zero + beta-one * x1 + beta-two * x2...." and a "variance = sigma squared".

Our then prof told us about "Method of Moments". He explained to us the idea of the "first moment", "second moment" ... "k-th moment" ... and how these moments are related to the Expected Values. For example, the "first moment" is the "mean" and the "variance" is a function of the first moment and the second moment.

He then told us about the Generalized Method of Moments (GMM), and that Regression Parameters can also be estimated using GMM. GMM requires you to solve a system of equations to get the parameter estimates and supposedly this system of equations is easier to solve than MLE, especially back in the day when computers were not as strong (I am not sure if this is true and why this is true).

So far, everything makes sense to me - but here is where I get mixed up:

  • Method of Moments requires you to set each "moment condition" equal to 0. I don't understand why this is necessary. In MLE, you need to take the derivatives and set them equal to zero, because the derivative corresponds to the "maximum point" on the (log) likelihood curve. But there appears to be no requirement for differentiation in Method of Moments - therefore, why do the moment conditions have to be set to 0? My guess is that this is just for "mathematical convenience" to make the system of equations solvable? (e.g. back in Linear Algebra class, we learned that a system of equations sometimes requires "conditions" to make it solvable, i.e. underdetermined system, over determined system)

  • Our prof mentioned that Generalized Method of Moments is closely related to OLS. He mentioned that Generalized Method of Moments is more "robust" than MLE, because Method of Moments does not require you to ASSUME a probability distribution (as is done in MLE). But you calculate moments by taking the "Expected Value" - and the "Expected Value" depends on a probability distribution! For example, the Expected Value of a Normal Distribution is "summation x_i/n" - but the Expected Value of a Exponential Distribution is "summation n_i/x". Therefore, why is the "Method of Moments" said NOT to depend on a probability distribution, and to be considered more robust than MLE - when it clearly does depend on a probability distribution?

  • 2
    $\begingroup$ This doesn’t address your main question, but it is worth pointing out that the OLS and MLE solutions coincide when the errors are $iid$ Gaussian (as is often assumed). $\endgroup$
    – Dave
    Sep 22 at 3:07
  • $\begingroup$ @ Dave: I was actually going to write that! Our prof mentioned this in the class today! $\endgroup$ Sep 22 at 3:09
  • 1
    $\begingroup$ What might be more robust depends on what kind of robustness you're looking at and how you specifically define your measure of it. Typically, in relation to estimators robustness would be defined to refer to some behavior of the estimator under some set of conditions (e.g. its response to gross errors), rather than whether its definition started from some assumed distribution. Any MLE - or indeed pretty much any estimator at all - might be declared to be "robust" by the same criterion as your professor used simply by forgetting that's how we obtained it. . . . ctd $\endgroup$
    – Glen_b
    Sep 22 at 8:03
  • 1
    $\begingroup$ ctd . . . Is the sample range robust as an estimator of scale? By the criterion "didn't assume a distribution" it appears that it must be, for all that it's fairly useless for that purpose for many distributions (and yet possibly quite useful for a few others). $\endgroup$
    – Glen_b
    Sep 22 at 8:06
  • 1
    $\begingroup$ Regarding In MLE, we <...> want to find out the "most likely" (i.e. highest probability) values of the "mean" and the "variance" (i.e. "parameters"): not quite – see this. The MLE maximizes the probability of the data, not the parameters. $\endgroup$ Sep 26 at 17:52


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy