# Linear Regression : Proving least squares model

My question is exactly similar to an earlier question Conditional on Gaussian, need clarification But I wasn't satisfied with the answer. I'll copy the problem statement :

I'm reading Andrew Ng's notes on machine learning, and on page 12 of this document, he makes a step in his proof that I'm trying to decipher:

Let $x=(1,x_1,x_2,⋯,x_n)^T$, a vector of variables, and $θ=(θ_0,θ_1,θ_2,⋯,θ_n)^T$, a vector of linear coefficients of those variables. Let's define $y$ as $$y_{i} = \theta^T x_{i} + \epsilon_i$$ where $\epsilon_i ∼ N(0, \sigma^2)$, that is $$p(\epsilon_i) = \frac{1}{\sqrt{2\pi\sigma}}\text{exp}\bigg(-\frac{\epsilon_i^2}{2\sigma^2}\bigg)$$

Next line says the following about conditional probability of y given x and coefficients θ, which are treated as deterministic: $$p(y_i|x_i;θ) = \frac{1}{\sqrt{2\pi\sigma}}\text{exp}\bigg(-\frac{(y_i - \theta^T x_i)^2}{2\sigma^2}\bigg)$$ Can someone help me see how we get this conditional distribution?

Now I understand that since $\epsilon_i$ is drawn from a gaussian, y will itself be a random variable drawn from a gaussian.

But it $y_i$ is a random variable with mean $\theta^T x_i$, so does it mean we have m (number of training samples) different gaussians and we draw 1 data point from each of m different gaussians and do a maximum liklihood estimation for that single data set for each of m gaussians, because this seems sort of unintuitive.

And also this only proves that $y_i$ is a gaussian not $y_i|x_i$. I dont understand why this has to hold?

• I think it is quite clear. $y_i|x_i$ is a gaussian is from $y_{i} = \theta^T x_{i} + \epsilon_i$, here both $x_i$ and $\theta$ are fixed, $\epsilon_i$ is the random variable. I think it is just notation, nothing special. – Deep North Dec 3 '17 at 22:39
• That's what i wrote but it implies yi is a gaussian nothing more. And as I wrote above we have m different gaussians and we take a single point from each of m gaussians, so this is not like normal likelihood estimations where we assume large data per random variable... – Naman Dec 4 '17 at 4:50

## 2 Answers

Now let's try to answer your question sequentially.

1) The equation $$y_{i} = \theta^{T}x_{i} + \epsilon_{i}$$ implies that $$y_{i}$$ is a Gaussian if $$x_{i}$$ is a constant. So if we rephrase it we can say that $$y_{i}$$ is a Gaussian for a given $$x_{i}$$ which implies that it's $$P(y_{i}|x_{i})$$ that's Gaussian and not $$P(y_{i})$$.

2) For the m different Gaussian's part, the thing that you are not taking into consideration is that we are trying to find a optimal value of $$\theta$$ among different possible values of $$\theta$$. For each of the value $$\theta$$ can take, we have m different Gaussian's which give value for $$P(y_{i}|x_{i})$$. Here $$y_{i}$$ is given to us we just need to find $$P(y_{i}|x_{i})$$ and we do not sample it out from the distribution. The value of $$\theta$$ that maximizes $$\prod_{i=1}^mP(y_{i}|x_{i})$$ will be termed as the maximum likelihood estimate as it's maximizing the likelihood of observing $$y_{i}$$ for $$x_{i}$$ i.e. $$P(y_{i}|x_{i})$$ for all $$x_{i}$$ in training set.

so does it mean we have m (number of training samples) different gaussians and we draw 1 data point from each of m different gaussians

In a way, yes. For an observation $$(x_i, y_i)$$, $$y_i$$ is considered to be a draw from a gaussian distribution with mean $$\theta^Tx_i$$. The mean of the gaussian is a function of the $$x_i$$.

And also this only proves that $$y_i$$ is a gaussian not $$y_i|x$$

No, this is wrong. If $$y_i$$ was gaussian, then the histogram of the outcome would look gaussian, but it doesn't have to in order to perform linear regression. See the example I make here.

Once I know $$x_i$$, then all the $$y_i$$ who have an associated $$x = x_i$$ are considered to be normal with mean $$\theta^Tx_i$$. We shorthand this by saying

$$y_i \vert x_i \sim \mathcal{N}(\theta^Tx_i, \sigma^2)$$

• "There is no distribution defined over all 𝑦𝑖s", yes, that is what I"m saying. OP appears to be arguing that the marginal distribution of the $y_i$ should be gaussian, and that is what I am saying they have wrong. – Demetri Pananos Jul 12 at 16:52
• Conditional distirbution of y (y conditioned on x) is gaussian. This is what the $y\vert x$ is getting at. The marginal distribution (e.g. the limit of the histogram of y) is not necessarily gaussian. Are we just disagreeing on what "marginal" means in this question, because it should be pretty unambiguous. – Demetri Pananos Jul 12 at 17:14