0
$\begingroup$

In the case of estimating regression coefficients, we all know that there are two common methods of estimating these coefficients:

  • Method 1: Ordinary Least Squares (OLS)

Ordinary Least Squares (OLS) estimates the values of the regression coefficients by minimizing the difference between values of the response variables and the predicted values of the response variable for a candidate set of regression coefficients. The final set of regression coefficients are selected based on which ever set minimizes the the above difference (i.e. the "error"). According to the Gauss-Markov Theorem, the estimates of the regression coefficients produced by OLS have certain desirable theoretical properties, such as being "unbiased".

  • Method 2: Maximum Likelihood Estimation (MLE)

On the other hand, Maximum Likelihood Estimation (MLE) estimates the values of the regression coefficient such that the final sets of estimates are most "likely" to reproduce the observed data (given a certain underlying probability distribution associated with the data and a class of model). It is worth noting that in many situations, the estimates provided by OLS and MLE tend to be similar.

My Question: Is there are theoretical relationship between OLS and MLE? MLE seems to have a probabilistic interpretation of the regression coefficients, whereas OLS seems to have a more geometric interpretation.

But is there any relationship between these two estimations techniques? For example, does OLS make any comments about the "probability of observing the data" - or does MLE make any comments about minimizing the error between predicted and observed values of the response variable?

Thanks!

Improve this question
$\endgroup$
4
  • $\begingroup$ Regarding Maximum Likelihood Estimation (MLE) estimates the values of the regression coefficient such that the final sets of estimates are most "likely" to reproduce the observed data, I am not sure about your formulation. The conditioning goes the other way: it is the data that is most likely given the parameters, not the parameters that are most likely given the data. See e.g. this. $\endgroup$
    – Richard Hardy
    Jan 5, 2022 at 5:36
  • $\begingroup$ The Gauss-Markov Theorem certainly does not imply that OLS is better than MLE. If you get the distributions approximately right, MLE can provide much more accurate estimates than OLS, biased or not. $\endgroup$
    – BigBendRegion
    Jan 5, 2022 at 13:23
  • $\begingroup$ I don't think I said that the Gauss-Markov Theorem implies that OLS is better than MLE - Gauss-Markov implies that the estimates produced by OLS tend to have certain "desirable properties", i.e. BLUE = Best Linear Unbiased Estimator. If the "Linear" aspect is not important, I believe there is something called the "Stein Estimator" which can be shown to have smaller variance than OLS, but will not necessarily be linear. $\endgroup$
    – stats_noob
    Jan 6, 2022 at 1:47
  • $\begingroup$ @stats555, It is not the linear aspect but the unbiased aspect. You get lower variance at the expense of some bias. $\endgroup$
    – Richard Hardy
    Jan 6, 2022 at 8:59

1 Answer 1

Reset to default
4
$\begingroup$

does MLE make any comments about minimizing the error between predicted and observed values of the response variable?

Consider a Gaussian GLM and consider the $\sigma^2$ as known. The negative log likelihood looks like

$$ y \vert x \sim \mathcal{N}(X\beta, \sigma^2) $$

$$ \ell(\beta; X) = \dfrac{1}{2\sigma} \sum (y-X\beta)^2 + C$$

Where $C$ does not depend on $\beta$ and hence does not contribute to the optimization problem. Does this expression look familiar? What if I write it as $X\beta = \hat{y}$?

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hello! Can you expand on this? $\endgroup$
    – stats_noob
    Sep 23, 2022 at 4:15
  • $\begingroup$ @MBA_Grad_Student_2022 Expand on what, in particular? $\endgroup$
    – Demetri Pananos
    Sep 23, 2022 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.