# Relationship between noise term ($\epsilon$) and MLE solution for Linear Regression Models

In Linear Regression models, given observed variables $x_1, x_2, x_3, ..., x_k$, unobserved (or predicted) variable $y$, and model parameters $\beta_0, \beta_1, \beta_2, \beta_3, ..., \beta_k$, it can be written as $$y = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k + \epsilon$$ where, $\epsilon$ is the noise term.

In vector notation the same thing can be written down as: $$\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \epsilon$$

Now this can be solved using maximum-likelihood estimation over $\beta$. That is: $$\hat{\beta} = argmax_{\beta} \Pr(\mathbf{y}|\mathbf{X}, \boldsymbol{\beta})$$

In order to specify $\Pr(\mathbf{y}|\mathbf{X}, \boldsymbol{\beta})$ mathematically, we need to make assumptions about the noise term $\epsilon$. A common assumption is that $\epsilon$ follows a Gaussian distribution with zero mean and variance $\sigma_{\epsilon}^{2}$, $$\epsilon \sim N(0, \sigma_{\epsilon}^{2})$$ This implies that the conditional probability density function of the output $Y$ for a given value of the input $X = x$ is given by $$\Pr(y|x, \beta) = N(y | \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2})$$

Now my question is, what has the distribution of $\epsilon$ (Normal distribution is this case), got to do with the distribution of the MLE?

In other words, why $\epsilon \sim N(0, \sigma_{\epsilon}^{2})$, also implies $$\Pr(y|x, \beta) = N(y | \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2})$$

• For the MLE, you can think the distribution of $Y_i$ is derived from $N(0,\sigma^2)$, i.e the distribution of $Y_i$ can be seen as a constant $X\beta$+$N(0,\sigma^2)$ then $Y_i$ has a $N(X\beta,\sigma^2)$ distribution from there you go to MLE. Feb 12, 2018 at 5:46
• I think the condistion probability is a little bit confusing. You may write $\Pr(y|x, \beta)$ imply $Y_i \sim N( \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2})$ Feb 12, 2018 at 5:48
• Yeah, this makes sense. Let me verify if I got it right. Assuming $\epsilon \sim N(0, \sigma_{\epsilon}^{2})$, gives us $Y_i \sim N(X\beta, \sigma_{\epsilon}^{2})$, which yields us the final conditional probability, that is: $\Pr(y|x, \beta) = N(y | \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2})$ Am I right? Feb 12, 2018 at 5:55
• I am not sure your last equation. If it stands for $Y_i$ has a $N( \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2})$ distribution then you are right. Feb 12, 2018 at 6:01
• Yes. Basically we are calculating the $\Pr({y})$ only, which is conditioned on $x$ and $\beta$. Now if $y$ has a distribution $N(y | \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2})$, then the last equation becomes clear. Feb 12, 2018 at 6:09

In other words, why $\epsilon \sim N(0, \sigma_{\epsilon}^{2})$, also implies $$\Pr(y|x, \beta) = N(y | \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2})$$

If $\epsilon \sim N(0, \sigma_{\epsilon}^{2})$

Then $y = X\beta + \epsilon \sim N(X\beta, \sigma_{\epsilon}^{2})$

Hence $y \sim N( \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2})$, which is basically the last line.

Note that $X\beta$ is a constant, and not a random variable in this setting, so $y$ is RV only because of $\epsilon$.

• Thank you for the explanation. And also for pointing out that $X\beta$ is a constant in this setting (since it's $y|x$). Feb 19, 2018 at 5:39
• Is 'x' assumed to be constant because it is conditioned upon? I am asking this because, in supervised learning, both x, and y are assumed to be coming from the joint distribution P(X, Y), which means both x and y are random variables. Dec 28, 2022 at 8:01

In linear regression with a normal error term the maximum-likelihood estimator for the coefficient vector is the ordinary least-squares (OLS) estimator. This can be written as:

\begin{equation} \begin{aligned} \hat{\boldsymbol{\beta}} = (\boldsymbol{X}^\text{T} \boldsymbol{X})^{-1} \boldsymbol{X}^\text{T} \boldsymbol{Y} &= (\boldsymbol{X}^\text{T} \boldsymbol{X})^{-1} \boldsymbol{X}^\text{T} (\boldsymbol{X} \boldsymbol{\beta} + \boldsymbol{\epsilon}) \\ &= \boldsymbol{\beta} + (\boldsymbol{X}^\text{T} \boldsymbol{X})^{-1} \boldsymbol{X}^\text{T} \boldsymbol{\epsilon}. \end{aligned} \end{equation}

This is a linear transformation of the error vector. Applying of the rules for linear transforms of a normal random vector, the joint distribution of the coefficient estimator and the error vector is:

$$\begin{bmatrix} \hat{\boldsymbol{\beta}} \\ \boldsymbol{\epsilon} \end{bmatrix} \sim \text{N} \left( \begin{bmatrix} \boldsymbol{\beta} \\ \boldsymbol{0} \end{bmatrix} , \sigma_{\epsilon}^2 \begin{bmatrix} (\boldsymbol{X}^\text{T} \boldsymbol{X})^{-1} & (\boldsymbol{X}^\text{T} \boldsymbol{X})^{-1} \boldsymbol{X}^\text{T} \\ \boldsymbol{X} (\boldsymbol{X}^\text{T} \boldsymbol{X})^{-1} & \boldsymbol{I} \end{bmatrix} \right) .$$

The joint distribution gives the full relationship between these random quantities. The error vector and coefficient estimator are correlated, with the correlation being fully determined by the design matrix $$\boldsymbol{X}$$.

• Thank you for providing with another perspective to the problem. :) Feb 19, 2018 at 5:41
• Should be $(X^\top X)^{-1}$ everywhere, no? Feb 7, 2022 at 7:26
• @R.Rayl: Yes, thank-you --- edited.
– Ben
Feb 7, 2022 at 7:48

The core is that this is a conditional distribution, i.e. we already known the input X.

So $$var(y) = var(X\beta + \epsilon)=var(\epsilon)$$ and $$E(y) = E(X\beta) + E[\epsilon]=X\beta$$.