Linear Regression : Proving least squares model My question is exactly similar to an earlier question Conditional on Gaussian, need clarification But I wasn't satisfied with the answer.
I'll copy the problem statement :


I'm reading Andrew Ng's notes on machine learning, and on page 12 of this document, he makes a step in his proof that I'm trying to decipher:
Let $x=(1,x_1,x_2,⋯,x_n)^T$, a vector of variables, and $θ=(θ_0,θ_1,θ_2,⋯,θ_n)^T$, a vector of linear coefficients of those variables. Let's define $y$ as $$y_{i} = \theta^T x_{i} + \epsilon_i$$ where $\epsilon_i ∼ N(0, \sigma^2)$, that is $$p(\epsilon_i) = \frac{1}{\sqrt{2\pi\sigma}}\text{exp}\bigg(-\frac{\epsilon_i^2}{2\sigma^2}\bigg)$$
Next line says the following about conditional probability of y given x and coefficients θ, which are treated as deterministic:
$$p(y_i|x_i;θ) = \frac{1}{\sqrt{2\pi\sigma}}\text{exp}\bigg(-\frac{(y_i - \theta^T x_i)^2}{2\sigma^2}\bigg)$$
Can someone help me see how we get this conditional distribution?


Now I understand that since $\epsilon_i$ is drawn from a gaussian, y will itself be a random variable drawn from a gaussian.
But it $y_i$ is a random variable with mean $\theta^T x_i$, so does it mean we have m (number of training samples) different gaussians and we draw 1 data point from each of m different gaussians and do a maximum liklihood estimation for that single data set for each of m gaussians, because this seems sort of unintuitive.
And also this only proves that $y_i$ is a gaussian not $y_i|x_i$. I dont understand why this has to hold?
 A: This one is quite simple: Conditional on the underlying explanatory variables and regression coefficients, the random variable $y_i$ is a function of the random variable $\epsilon_i$.  We can denote this function as:
$$y_i = h_i(\epsilon_i)
\quad \quad \quad \quad \quad 
h_i(e) = \theta^\text{T} x_i + e,$$
and we denote its inverse function $g_i = h_i^{-1}$ as:
$$y_i = h_i(\epsilon_i)
\quad \quad \quad \quad \quad 
g_i(y) = y-\theta^\text{T} x_i.$$
Now, we can easily see that the Jacobian of this transformation is one, so using the rules for the density of a transformed random variable, we then have:
$$\begin{aligned}
f(y_i|x_i,\theta) 
&= f(\epsilon_i = g_i(y_i)|x_i,\theta) \times \Big| \frac{d g}{d y} (y_i) \Big| \\[6pt]
&= f(\epsilon_i = g_i(y_i)) \\[6pt]
&= \frac{1}{\sqrt{2 \pi \sigma^2}} \cdot \text{exp} \bigg( - \frac{g_i(y_i)^2}{2\sigma^2} \bigg) \\[6pt]
&= \frac{1}{\sqrt{2 \pi \sigma^2}} \cdot \text{exp} \bigg( - \frac{(y_i - \theta^T x_i)^2}{2\sigma^2} \bigg). \\[6pt]
\end{aligned}$$
(Note that you have written the constant term for the normal density function incorrectly.)  Now, all of this is just for a single data point, and in a regression model you would speficy that this holds for data points $i=1,...,n$, which would lead you to a likelihood function for the whole sample.
A: Now let's try to answer your question sequentially.
1) The equation $y_{i} = \theta^{T}x_{i} + \epsilon_{i}$ implies that $y_{i}$ is a Gaussian if $x_{i}$ is a constant. So if we rephrase it we can say that $y_{i}$ is a Gaussian for a given $x_{i}$ which implies that it's $P(y_{i}|x_{i})$ that's Gaussian and not $P(y_{i})$.
2) For the m different Gaussian's part, the thing that you are not taking into consideration is that we are trying to find a optimal value of $\theta$ among different possible values of $\theta$. For each of the value $\theta$ can take, we have m different Gaussian's which give value for $P(y_{i}|x_{i})$. Here $y_{i}$ is given to us we just need to find $P(y_{i}|x_{i})$ and we do not sample it out from the distribution. The value of $\theta$ that maximizes $\prod_{i=1}^mP(y_{i}|x_{i})$ will be termed as the maximum likelihood estimate as it's maximizing the likelihood of observing $y_{i}$ for  $x_{i}$ i.e. $P(y_{i}|x_{i})$ for all $x_{i}$ in training set.
A: 
so does it mean we have m (number of training samples) different gaussians and we draw 1 data point from each of m different gaussians 

In a way, yes.  For an observation $(x_i, y_i)$, $y_i$ is considered to be a draw from a gaussian distribution with mean $\theta^Tx_i$.  The mean of the gaussian is a function of the $x_i$.

And also this only proves that $y_i$ is a gaussian not $y_i|x$

No, this is wrong.  If $y_i$ was gaussian, then the histogram of the outcome would look gaussian, but it doesn't have to in order to perform linear regression. See the example I make here.
Once I know $x_i$, then all the $y_i$ who have an associated $x = x_i$ are considered to be normal with mean $\theta^Tx_i$.  We shorthand this by saying
$$ y_i \vert x_i \sim \mathcal{N}(\theta^Tx_i, \sigma^2)$$
