Linear Regression : Proving least squares model

My question is exactly similar to an earlier question Conditional on Gaussian, need clarification But I wasn't satisfied with the answer. I'll copy the problem statement :

I'm reading Andrew Ng's notes on machine learning, and on page 12 of this document, he makes a step in his proof that I'm trying to decipher:

Let $$x=(1,x_1,x_2,⋯,x_n)^T$$, a vector of variables, and $$θ=(θ_0,θ_1,θ_2,⋯,θ_n)^T$$, a vector of linear coefficients of those variables. Let's define $$y$$ as $$y_{i} = \theta^T x_{i} + \epsilon_i$$ where $$\epsilon_i ∼ N(0, \sigma^2)$$, that is $$p(\epsilon_i) = \frac{1}{\sqrt{2\pi\sigma}}\text{exp}\bigg(-\frac{\epsilon_i^2}{2\sigma^2}\bigg)$$

Next line says the following about conditional probability of y given x and coefficients θ, which are treated as deterministic: $$p(y_i|x_i;θ) = \frac{1}{\sqrt{2\pi\sigma}}\text{exp}\bigg(-\frac{(y_i - \theta^T x_i)^2}{2\sigma^2}\bigg)$$ Can someone help me see how we get this conditional distribution?

Now I understand that since $$\epsilon_i$$ is drawn from a gaussian, y will itself be a random variable drawn from a gaussian.

But it $$y_i$$ is a random variable with mean $$\theta^T x_i$$, so does it mean we have m (number of training samples) different gaussians and we draw 1 data point from each of m different gaussians and do a maximum liklihood estimation for that single data set for each of m gaussians, because this seems sort of unintuitive.

And also this only proves that $$y_i$$ is a gaussian not $$y_i|x_i$$. I dont understand why this has to hold?

• I think it is quite clear. $y_i|x_i$ is a gaussian is from $y_{i} = \theta^T x_{i} + \epsilon_i$, here both $x_i$ and $\theta$ are fixed, $\epsilon_i$ is the random variable. I think it is just notation, nothing special. – Deep North Dec 3 '17 at 22:39

so does it mean we have m (number of training samples) different gaussians and we draw 1 data point from each of m different gaussians

In a way, yes. For an observation $$(x_i, y_i)$$, $$y_i$$ is considered to be a draw from a gaussian distribution with mean $$\theta^Tx_i$$. The mean of the gaussian is a function of the $$x_i$$.

And also this only proves that $$y_i$$ is a gaussian not $$y_i|x$$

No, this is wrong. If $$y_i$$ was gaussian, then the histogram of the outcome would look gaussian, but it doesn't have to in order to perform linear regression. See the example I make here.

Once I know $$x_i$$, then all the $$y_i$$ who have an associated $$x = x_i$$ are considered to be normal with mean $$\theta^Tx_i$$. We shorthand this by saying

$$y_i \vert x_i \sim \mathcal{N}(\theta^Tx_i, \sigma^2)$$

• "There is no distribution defined over all 𝑦𝑖s", yes, that is what I"m saying. OP appears to be arguing that the marginal distribution of the $y_i$ should be gaussian, and that is what I am saying they have wrong. – Demetri Pananos Jul 12 '19 at 16:52
• Conditional distirbution of y (y conditioned on x) is gaussian. This is what the $y\vert x$ is getting at. The marginal distribution (e.g. the limit of the histogram of y) is not necessarily gaussian. Are we just disagreeing on what "marginal" means in this question, because it should be pretty unambiguous. – Demetri Pananos Jul 12 '19 at 17:14

This one is quite simple: Conditional on the underlying explanatory variables and regression coefficients, the random variable $$y_i$$ is a function of the random variable $$\epsilon_i$$. We can denote this function as:

$$y_i = h_i(\epsilon_i) \quad \quad \quad \quad \quad h_i(e) = \theta^\text{T} x_i + e,$$

and we denote its inverse function $$g_i = h_i^{-1}$$ as:

$$y_i = h_i(\epsilon_i) \quad \quad \quad \quad \quad g_i(y) = y-\theta^\text{T} x_i.$$

Now, we can easily see that the Jacobian of this transformation is one, so using the rules for the density of a transformed random variable, we then have:

\begin{aligned} f(y_i|x_i,\theta) &= f(\epsilon_i = g_i(y_i)|x_i,\theta) \times \Big| \frac{d g}{d y} (y_i) \Big| \\[6pt] &= f(\epsilon_i = g_i(y_i)) \\[6pt] &= \frac{1}{\sqrt{2 \pi \sigma^2}} \cdot \text{exp} \bigg( - \frac{g_i(y_i)^2}{2\sigma^2} \bigg) \\[6pt] &= \frac{1}{\sqrt{2 \pi \sigma^2}} \cdot \text{exp} \bigg( - \frac{(y_i - \theta^T x_i)^2}{2\sigma^2} \bigg). \\[6pt] \end{aligned}

(Note that you have written the constant term for the normal density function incorrectly.) Now, all of this is just for a single data point, and in a regression model you would speficy that this holds for data points $$i=1,...,n$$, which would lead you to a likelihood function for the whole sample.

1) The equation $$y_{i} = \theta^{T}x_{i} + \epsilon_{i}$$ implies that $$y_{i}$$ is a Gaussian if $$x_{i}$$ is a constant. So if we rephrase it we can say that $$y_{i}$$ is a Gaussian for a given $$x_{i}$$ which implies that it's $$P(y_{i}|x_{i})$$ that's Gaussian and not $$P(y_{i})$$.
2) For the m different Gaussian's part, the thing that you are not taking into consideration is that we are trying to find a optimal value of $$\theta$$ among different possible values of $$\theta$$. For each of the value $$\theta$$ can take, we have m different Gaussian's which give value for $$P(y_{i}|x_{i})$$. Here $$y_{i}$$ is given to us we just need to find $$P(y_{i}|x_{i})$$ and we do not sample it out from the distribution. The value of $$\theta$$ that maximizes $$\prod_{i=1}^mP(y_{i}|x_{i})$$ will be termed as the maximum likelihood estimate as it's maximizing the likelihood of observing $$y_{i}$$ for $$x_{i}$$ i.e. $$P(y_{i}|x_{i})$$ for all $$x_{i}$$ in training set.