Suppose we want a regression of a function $f(x)$. Suppose $r = f(x) + \epsilon$. Why can we suppose that $\epsilon \sim \mathcal{N}(0,\sigma^2)$? What is the advantage of such supposition?

  • 2
    $\begingroup$ Welcome to Cross Validated! What do you mean by "why can we suppose?" We can suppose whatever we want for any reason. Your second question about why we might want to make this assumption makes sense, though it is addressed elsewhere on Cross Validated. $\endgroup$ – Dave Sep 11 at 17:56
  • 1
    $\begingroup$ @Dave I am just not sure to understand that supposition. I am following an introduction to AI course and the teacher shoot that out of no where, but I am not even sure to understand why he did that. $\endgroup$ – J.Doe Sep 11 at 18:02
  • $\begingroup$ stats.stackexchange.com/questions/148803/… ; stats.stackexchange.com/questions/395011/… provide some comment $\endgroup$ – user2957945 Sep 11 at 18:50
  • $\begingroup$ This and related topics have been discussed in dozens of threads on Cross Validated... $\endgroup$ – Richard Hardy Sep 11 at 19:00

This assumption has two reasons.

First, it is reasonable to assume a normal distribution for the error. We make this assumption because if you have many random variables that are influencing the error independently and additively the distribution of the resulting random variable follows the normal distribution.

Another advantage of this assumption is rooted in parameter estimation for linear regression. If we assume that $f(\mathbf{x}) = \mathbf{w}^T\mathbf{x}$, we can rewrite the residual of observation $i$ as

$$\varepsilon_i = r_i - f(\mathbf{x}_i).$$

If $\varepsilon \sim \mathcal{N}(0, \sigma^2)$, then we know that for observation $i$ we have

$$p(\varepsilon_i) = \dfrac{1}{\sqrt{2\pi \sigma^2}}\exp\left[-0.5(\varepsilon_i - 0)^2/\sigma^2\right]$$

Swichting from $\varepsilon_i$ to $\mathbf{x}_i$ and $r_i$ will result in $$p(\mathbf{x}_i,r_i|\mathbf{w}) = \dfrac{1}{\sqrt{2\pi \sigma^2}}\exp\left[-0.5(r_i - \mathbf{w}^T\mathbf{x}_i)^2/\sigma^2\right].$$

If we assume that our errors are independent, then we can express the likelihood for observing the data $\mathcal{D}=\{(\mathbf{x}_1,r_i),\ldots,(\mathbf{x}_N, r_N) \}$ as

$$L(\mathcal{D}|\mathbf{w}) = \prod_{n=1}^N\dfrac{1}{\sqrt{2\pi \sigma^2}}\exp\left[-0.5(r_n - \mathbf{w}^T\mathbf{x}_n)^2/\sigma^2\right].$$

The log-likelihood of this expression is given as $$\log L(\mathcal{D}|\mathbf{w}) = \log \left[\dfrac{1}{\sqrt{2\pi \sigma^2}}\right]^N -\dfrac{1}{2\sigma^2}\sum_{n=1}^N\left[r_n - \mathbf{w}^T\mathbf{x}_n\right]^2.$$

If we want to maximize the log-likelihood (maximizes the likelihood of observing the data $\mathcal{D}$) we need to minimize (note the negative sign of the sum)

$$\sum_{n=1}^N\left[r_n - \mathbf{w}^T\mathbf{x}_n\right]^2.$$

But this is the sum of squared errors that we minimize in the standard case of multiple linear regression.

| cite | improve this answer | |
  • 1
    $\begingroup$ You put $x_i$ and $y_i$ in $p(x_i, y_i | w)$. You probably meant $p(x_i, r_i | w)$ $\endgroup$ – J.Doe Sep 11 at 23:14
  • $\begingroup$ Thank you for pointing this out. I should have redeclared the target in the first line. $\endgroup$ – MachineLearner Sep 11 at 23:16
  • $\begingroup$ Very clear as answer! I understand your point! :D $\endgroup$ – J.Doe Sep 11 at 23:16

Not the answer you're looking for? Browse other questions tagged or ask your own question.