In Linear Regression models, given observed variables $x_1, x_2, x_3, ..., x_k$, unobserved (or predicted) variable $y$, and model parameters $\beta_0, \beta_1, \beta_2, \beta_3, ..., \beta_k$, it can be written as $$ y = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k + \epsilon $$ where, $\epsilon$ is the noise term.

In vector notation the same thing can be written down as: $$ \mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \epsilon $$

Now this can be solved using maximum-likelihood estimation over $\beta$. That is: $$ \hat{\beta} = argmax_{\beta} \Pr(\mathbf{y}|\mathbf{X}, \boldsymbol{\beta}) $$

Now I read somewhere that

In order to specify $\Pr(\mathbf{y}|\mathbf{X}, \boldsymbol{\beta})$ mathematically, we need to make assumptions about the noise term $\epsilon$. A common assumption is that $\epsilon$ follows a Gaussian distribution with zero mean and variance $\sigma_{\epsilon}^{2}$, $$ \epsilon \sim N(0, \sigma_{\epsilon}^{2}) $$ This implies that the conditional probability density function of the output $Y$ for a given value of the input $X = x$ is given by $$ \Pr(y|x, \beta) = N(y | \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2}) $$

Now my question is, what has the distribution of $\epsilon$ (Normal distribution is this case), got to do with the distribution of the MLE?

In other words, why $\epsilon \sim N(0, \sigma_{\epsilon}^{2})$, also implies $$ \Pr(y|x, \beta) = N(y | \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2}) $$

Thank you in advance.

  • $\begingroup$ For the MLE, you can think the distribution of $Y_i$ is derived from $N(0,\sigma^2)$, i.e the distribution of $Y_i$ can be seen as a constant $X\beta$+$N(0,\sigma^2)$ then $Y_i$ has a $N(X\beta,\sigma^2)$ distribution from there you go to MLE. $\endgroup$ – Deep North Feb 12 '18 at 5:46
  • $\begingroup$ I think the condistion probability is a little bit confusing. You may write $\Pr(y|x, \beta)$ imply $Y_i \sim N( \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2})$ $\endgroup$ – Deep North Feb 12 '18 at 5:48
  • $\begingroup$ Yeah, this makes sense. Let me verify if I got it right. Assuming $\epsilon \sim N(0, \sigma_{\epsilon}^{2})$, gives us $Y_i \sim N(X\beta, \sigma_{\epsilon}^{2})$, which yields us the final conditional probability, that is: $\Pr(y|x, \beta) = N(y | \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2})$ Am I right? $\endgroup$ – Koustav Feb 12 '18 at 5:55
  • $\begingroup$ I am not sure your last equation. If it stands for $Y_i$ has a $N( \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2})$ distribution then you are right. $\endgroup$ – Deep North Feb 12 '18 at 6:01
  • $\begingroup$ Yes. Basically we are calculating the $\Pr({y})$ only, which is conditioned on $x$ and $\beta$. Now if $y$ has a distribution $N(y | \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2})$, then the last equation becomes clear. $\endgroup$ – Koustav Feb 12 '18 at 6:09

In other words, why $\epsilon \sim N(0, \sigma_{\epsilon}^{2})$, also implies $$ \Pr(y|x, \beta) = N(y | \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2}) $$

If $\epsilon \sim N(0, \sigma_{\epsilon}^{2})$

Then $y = X\beta + \epsilon \sim N(X\beta, \sigma_{\epsilon}^{2})$

Hence $y \sim N( \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2})$, which is basically the last line.

Note that $X\beta$ is a constant, and not a random variable in this setting, so $y$ is RV only because of $\epsilon$.

  • $\begingroup$ Thank you for the explanation. And also for pointing out that $X\beta$ is a constant in this setting (since it's $y|x$). $\endgroup$ – Koustav Feb 19 '18 at 5:39

In linear regression with a normal error term the maximum-likelihood estimator for the coefficient vector is the ordinary least-squares (OLS) estimator. This can be written as:

$$\begin{equation} \begin{aligned} \hat{\boldsymbol{\beta}} = (\boldsymbol{X}^\text{T} \boldsymbol{X}) \boldsymbol{X}^\text{T} \boldsymbol{Y} &= (\boldsymbol{X}^\text{T} \boldsymbol{X}) \boldsymbol{X}^\text{T} (\boldsymbol{X} \boldsymbol{\beta} + \boldsymbol{\epsilon}) \\ &= \boldsymbol{\beta} + (\boldsymbol{X}^\text{T} \boldsymbol{X}) \boldsymbol{X}^\text{T} \boldsymbol{\epsilon}. \end{aligned} \end{equation}$$

This is a linear transformation of the error vector. Applying of the rules for linear transforms of a normal random vector, the joint distribution of the coefficient estimator and the error vector is:

$$\begin{bmatrix} \hat{\boldsymbol{\beta}} \\ \boldsymbol{\epsilon} \end{bmatrix} \sim \text{N} \left( \begin{bmatrix} \boldsymbol{\beta} \\ \boldsymbol{0} \end{bmatrix} , \sigma_{\epsilon}^2 \begin{bmatrix} \boldsymbol{X}^\text{T} \boldsymbol{X} & (\boldsymbol{X}^\text{T} \boldsymbol{X}) \boldsymbol{X}^\text{T} \\ \boldsymbol{X} (\boldsymbol{X}^\text{T} \boldsymbol{X}) & \boldsymbol{I} \end{bmatrix} \right) .$$

The joint distribution gives the full relationship between these random quantities. The error vector and coefficient estimator are correlated, with the correlation being fully determined by the design matrix $\boldsymbol{X}$.

  • $\begingroup$ Thank you for providing with another perspective to the problem. :) $\endgroup$ – Koustav Feb 19 '18 at 5:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.