Relation between OLS, MM and ML What is the relation between OLS, MM (method of moments) and ML (maximum likelihood)? During my studies, the three concepts got taught completely separated from each other. However, they seem to be strongly related. Could anyone shed some light on this in basic/broad terms?
 A: These are different methods to estimate parameters, however they are related. The Gaussian (normal) distribution in particular has $(x-a)^2$ as a term in the loglikelihood, which means that maximising the likelihood over $a$ for independent observations (involving a product of the densities that becomes a sum after taking the log) amounts to minimising the sum of squares. In other words, the ordinary least squares (OLS) estimator will be the same as maximum likelihood, same in regression with Gaussian errors.
For other distributions they are in general not the same. Least squares has the direct interpretation that the estimator is chosen so that it approximates all the data well in the sense of making squared distances small, which may make sense even for non-normal distributions, however in those cases maximising the likelihood will be something different, normally with better statistical properties (as observations are used in a way optimal for the model rather than following a general principle).
In the Gaussian case, the method of moments also will give the same estimators as maximum likelihood for both mean and variance, however it is a different principle and in general it will give different estimators. It is more mathematically motivated, choosing estimators that are in line with the observed moments, rather than approximating the data (like OLS) or trying to be optimal at the model regarding the likelihood.
The Gaussian distribution is very popular and here the methods coincide; actually some people would take the OLS principle as being motivated by the Gaussian likelihood and would say Gaussian distributions have to be assumed when computing OLS estimators (which means they are also ML), but this is not strictly true, the OLS principle can be motivated also for other situations, but may lose some quality there.
