1
$\begingroup$

What are the model assumptions necessary for p(y|x,w) in order for the MLE estimate to provide the OLS loss function?

I know that MLE applied to linear regression gives OLS, I'm just not sure what assumptions, if any, must hold for this to be the case.

$\endgroup$
1
  • 4
    $\begingroup$ This sounds like it’s for a class. Please add the self-stud tag, read its wiki, and tell us what progress you’ve made so far. $\endgroup$ – Dave Mar 17 at 0:36
1
$\begingroup$

OLS relations with MLE

Maximum likelihood estimation (MLE) can be performed when the distribution of the error terms is known to belong to a certain parametric family ƒθ of probability distributions, and assuming randoms sampling. When ƒθ is a normal distribution with zero mean and variance θ, the resulting estimate is identical to the OLS estimate. GLS estimates are maximum likelihood estimates when ε follows a multivariate normal distribution with a known covariance matrix. Wikipedia

The goal of maximum likelihood estimation is to make inferences about the population that is most likely to have generated the sample, specifically the joint probability distribution of the random variables $\left\{ y_{1}, y_{2}, \ldots \right\}$, not necessarily independent and identically distributed.

OLS can be estimated without making full distributional assumption. Instead, we focused primarily on zero-covariance and zero-conditional-mean assumptions, and secondarily on assumptions about conditional variances and covariances. These assumptions were sufficient for obtaining consistent, asymptotically normal estimators, some of which were shown to be efficient within certain classes of estimators. As Wooldridge 2010 Page 385.

Here MLE of the OLS in detail https://www.le.ac.uk/users/dsgp1/COURSES/MATHSTAT/13mlreg.pdf

Pros and Cons

Pros

  1. MLE as the unifying theme: most models are estimated by maximum likelihood. As Wooldridge 2010 Page 385.
  2. Efficient Properties: it is generally the most efficient estimation procedure in the class of estimators that use information on the distribution of the endogenous variables given the exogenous variables. As Wooldridge 2010 Page 385.

Cons

  1. Efficiency usually comes at the price of nonrobustness, and this is certainly the case for maximum likelihood. Maximum likelihood estimators are generally inconsistent if some part of the specified distribution is misspecified. As Wooldridge 2010 Page 385.

References

  1. Greene, William H. "Econometric analysis 4th edition." International edition, New Jersey: Prentice Hall (2000): 201-215. https://spu.fem.uniag.sk/cvicenia/ksov/obtulovic/Mana%C5%BE.%20%C5%A1tatistika%20a%20ekonometria/EconometricsGREENE.pdf
  2. Wooldridge, Jeffrey M. Econometric analysis of cross section and panel data. MIT press, 2010. https://jrvargas.files.wordpress.com/2011/01/wooldridge_j-_2002_econometric_analysis_of_cross_section_and_panel_data.pdf
$\endgroup$
1
  • 1
    $\begingroup$ Note this is a self-study question, so the answer should be phrased accordingly. $\endgroup$ – Richard Hardy Mar 21 at 14:22
-2
$\begingroup$

Basically, five assumptions of linear regression must be satisfied:

  1. Linearity
  2. Little or no multicollinearity
  3. Homoscedasticity
  4. Independence
  5. Normality

Suppose $$Y = X\beta + \epsilon, \epsilon \sim N(0_n,\sigma^2I_n)$$ the probability density function of Y: $$f(Y,\beta)=\frac{1}{\sqrt{(2\pi)^n|\Sigma|}} \exp\left(-\frac{1}{2}({ Y}-{ X\beta})^T {\Sigma}^{-1}({ Y}-{X\beta}) \right)$$ The MLE for $\beta$:
$$\hat{\beta}_{MLE} = (X^TX)^{-1}X^TY$$ The residual sum of square is: $$S(\beta) = (Y-X\beta)^T(Y-X\beta)$$
The OLS estimator for $\beta$:
$$\hat{\beta}_{OLS}= (X^TX)^{-1}X^TY$$ If any assumption breaks, for example, $\mathbf{X}$ has perfect multicollinearity, then $\mathbf{X^TX}$ will not be full rank, which makes $\mathbf{X^TX}$ non-invertible.

$\endgroup$
1
  • 2
    $\begingroup$ Note this is a self-study question, so the answer should be phrased accordingly. Also, Little or no multicollinearity is incorrect; only perfect multicollinearity is a problem. Also, If any assumption breaks ... then $X^TX$ will not be full rank is incorrect. It seems you started your sentence with one thing in mind, but ended with another. $\endgroup$ – Richard Hardy Mar 21 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.