Linear Regression : Proving least squares model

Question

My question is exactly similar to an earlier question Conditional on Gaussian, need clarification But I wasn't satisfied with the answer. I'll copy the problem statement :

---

I'm reading Andrew Ng's notes on machine learning, and on page 12 of this document, he makes a step in his proof that I'm trying to decipher:

Let $x=(1,x_1,x_2,⋯,x_n)^T$, a vector of variables, and $θ=(θ_0,θ_1,θ_2,⋯,θ_n)^T$, a vector of linear coefficients of those variables. Let's define $y$ as $$y_{i} = \theta^T x_{i} + \epsilon_i$$ where $\epsilon_i ∼ N(0, \sigma^2)$, that is $$p(\epsilon_i) = \frac{1}{\sqrt{2\pi\sigma}}\text{exp}\bigg(-\frac{\epsilon_i^2}{2\sigma^2}\bigg)$$

Next line says the following about conditional probability of y given x and coefficients θ, which are treated as deterministic: $$p(y_i|x_i;θ) = \frac{1}{\sqrt{2\pi\sigma}}\text{exp}\bigg(-\frac{(y_i - \theta^T x_i)^2}{2\sigma^2}\bigg)$$ Can someone help me see how we get this conditional distribution?

---

Now I understand that since $\epsilon_i$ is drawn from a gaussian, y will itself be a random variable drawn from a gaussian.

But it $y_i$ is a random variable with mean $\theta^T x_i$, so does it mean we have m (number of training samples) different gaussians and we draw 1 data point from each of m different gaussians and do a maximum liklihood estimation for that single data set for each of m gaussians, because this seems sort of unintuitive.

And also this only proves that $y_i$ is a gaussian not $y_i|x_i$. I dont understand why this has to hold?

Answer 1

This one is quite simple: Conditional on the underlying explanatory variables and regression coefficients, the random variable $y_i$ is a function of the random variable $\epsilon_i$. We can denote this function as:

$$y_i = h_i(\epsilon_i) \quad \quad \quad \quad \quad h_i(e) = \theta^\text{T} x_i + e,$$

and we denote its inverse function $g_i = h_i^{-1}$ as:

$$y_i = h_i(\epsilon_i) \quad \quad \quad \quad \quad g_i(y) = y-\theta^\text{T} x_i.$$

Now, we can easily see that the Jacobian of this transformation is one, so using the rules for the density of a transformed random variable, we then have:

$$\begin{aligned} f(y_i|x_i,\theta) &= f(\epsilon_i = g_i(y_i)|x_i,\theta) \times \Big| \frac{d g}{d y} (y_i) \Big| \\[6pt] &= f(\epsilon_i = g_i(y_i)) \\[6pt] &= \frac{1}{\sqrt{2 \pi \sigma^2}} \cdot \text{exp} \bigg( - \frac{g_i(y_i)^2}{2\sigma^2} \bigg) \\[6pt] &= \frac{1}{\sqrt{2 \pi \sigma^2}} \cdot \text{exp} \bigg( - \frac{(y_i - \theta^T x_i)^2}{2\sigma^2} \bigg). \\[6pt] \end{aligned}$$

(Note that you have written the constant term for the normal density function incorrectly.) Now, all of this is just for a single data point, and in a regression model you would speficy that this holds for data points $i=1,...,n$, which would lead you to a likelihood function for the whole sample.

Answer 2

Now let's try to answer your question sequentially.

1) The equation $y_{i} = \theta^{T}x_{i} + \epsilon_{i}$ implies that $y_{i}$ is a Gaussian if $x_{i}$ is a constant. So if we rephrase it we can say that $y_{i}$ is a Gaussian for a given $x_{i}$ which implies that it's $P(y_{i}|x_{i})$ that's Gaussian and not $P(y_{i})$.

2) For the m different Gaussian's part, the thing that you are not taking into consideration is that we are trying to find a optimal value of $\theta$ among different possible values of $\theta$. For each of the value $\theta$ can take, we have m different Gaussian's which give value for $P(y_{i}|x_{i})$. Here $y_{i}$ is given to us we just need to find $P(y_{i}|x_{i})$ and we do not sample it out from the distribution. The value of $\theta$ that maximizes $\prod_{i=1}^mP(y_{i}|x_{i})$ will be termed as the maximum likelihood estimate as it's maximizing the likelihood of observing $y_{i}$ for $x_{i}$ i.e. $P(y_{i}|x_{i})$ for all $x_{i}$ in training set.

Answer 3

so does it mean we have m (number of training samples) different gaussians and we draw 1 data point from each of m different gaussians

In a way, yes. For an observation $(x_i, y_i)$, $y_i$ is considered to be a draw from a gaussian distribution with mean $\theta^Tx_i$. The mean of the gaussian is a function of the $x_i$.

And also this only proves that $y_i$ is a gaussian not $y_i|x$

No, this is wrong. If $y_i$ was gaussian, then the histogram of the outcome would look gaussian, but it doesn't have to in order to perform linear regression. See the example I make here.

Once I know $x_i$, then all the $y_i$ who have an associated $x = x_i$ are considered to be normal with mean $\theta^Tx_i$. We shorthand this by saying

$$ y_i \vert x_i \sim \mathcal{N}(\theta^Tx_i, \sigma^2)$$