In the case of estimating regression coefficients, we all know that there are two common methods of estimating these coefficients:
- Method 1: Ordinary Least Squares (OLS)
Ordinary Least Squares (OLS) estimates the values of the regression coefficients by minimizing the difference between values of the response variables and the predicted values of the response variable for a candidate set of regression coefficients. The final set of regression coefficients are selected based on which ever set minimizes the the above difference (i.e. the "error"). According to the Gauss-Markov Theorem, the estimates of the regression coefficients produced by OLS have certain desirable theoretical properties, such as being "unbiased".
- Method 2: Maximum Likelihood Estimation (MLE)
On the other hand, Maximum Likelihood Estimation (MLE) estimates the values of the regression coefficient such that the final sets of estimates are most "likely" to reproduce the observed data (given a certain underlying probability distribution associated with the data and a class of model). It is worth noting that in many situations, the estimates provided by OLS and MLE tend to be similar.
My Question: Is there are theoretical relationship between OLS and MLE? MLE seems to have a probabilistic interpretation of the regression coefficients, whereas OLS seems to have a more geometric interpretation.
But is there any relationship between these two estimations techniques? For example, does OLS make any comments about the "probability of observing the data" - or does MLE make any comments about minimizing the error between predicted and observed values of the response variable?