Maximum Likelihood with Least Squared Error

Question

In the book Tom Mitchell - Machine Learning, while deriving Least Squared Error from maximum likelihood, the author considers the training dataset of the form: $<x_i, d_i>$ where: $$d_i = f(x_i) + e_i$$ Here, $f(x_i)$ is the noise free value of the target function and $e_i$ is the random variable representing noise, which is distributed according to normal distribution with $0$ mean.

The author then says that given the noise $e_i$ obeys a Normal distribution with 0 mean and an unknown variance $\sigma^2$, each $d_i$ must also obey a Normal distribution with variance $\sigma^2$, centered around the true target value $f(x_i)$.

Can anyone please explain that if the error $e_i$ is Normally distributed, then why should $d_i$ also be Normally distributed ?

Answer 1

Let $z \sim \mathcal{N}(\mu, \sigma)$. Then

$$ \dfrac{z-\mu}{\sigma} \sim \mathcal{N}(0,1)$$

Conversely, if $x \sim \mathcal{N}(0,1)$, then

$$ \mu + \sigma x \sim \mathcal{N}(\mu,\sigma)$$

The noise is normal $e_i \sim \mathcal{N}(0,\sigma)$, so if I add some noiseless constant to this random variable, the mean changes

$$ f(x_i) + e_i = d_i \sim \mathcal{N}(f(x_i), \sigma)$$

EDIT: There is a slight abuse of terminology in most regression text. Note that $d_i$ corresponds to observations of $x_i$. So this means the conditional distribution of the outcome is normal, not the marginal. Mathematically

$$ d_i \vert x_i \sim \mathcal{N}(f(x_i), \sigma)$$

Answer 2

Brief expansion of user Christoph Hanck's comment:

The measurements $x_i$ are assumed to be known exactly.* Under this assumption, $f(x_i)$ is distributed like a normal distribution with mean $f(x_i)$ and variance $0$. If $e_i\sim\mathcal{N}(0, \sigma^2)$, it follows that $$\underbrace{d_i}_{\sim\mathcal{N}(f(x_i), \sigma^2)} = \underbrace{f(x_i)}_{\sim\mathcal{N}(f(x_i), 0)} + \underbrace{e_i}_{\sim\mathcal{N}(0, \sigma^2)}.$$

*This assumption can of course be criticized, see, e.g., this question.