# Conditional independence statements for probabilistic motivation for linear regression

So the motivation for using the squared loss in linear regression can be written as the following (I think):

Assume $$\{(\mathbf{x}_i, y_i) \mid i = 1, \dots n\}$$ are repeated independent samples from random variables $$\mathbf{X}$$ and $$Y$$ respectively. With $$Y \mid \mathbf{X} \sim N(f(\mathbf{x};\mathbf{w}), \sigma^2)$$.

We then can write that:

$$p(y_1, \dots y_n \mid \mathbf{x}_1, \dots \mathbf{x}_n; \mathbf{w}, \sigma) = \frac{\prod_{i=1}^{n}{p(\mathbf{x}_i,y_i;\mathbf{w}, \sigma)}}{\prod_{i=1}^{n}{p(\mathbf{x}_i;\mathbf{w}, \sigma)}} \\ = \prod_{i=1}^{n}{p(y_i \mid \mathbf{x}_i;\mathbf{w}, \sigma)}$$

And therefore under the distributional assumption the log-likelihood is given by:

\begin{align} \log(p(y_1, \dots y_n \mid \mathbf{x}_1, \dots \mathbf{x}_n; \mathbf{w}, \sigma)) = C + \frac{\sum_{i=1}^{n}{(y_i - f(\mathbf{x}_i;\mathbf{w}))^2}}{2\sigma^2} \end{align}

And hence the MLE of $$\mathbf{w}$$ is given by solving the least squares problem.

Above we assumed $$(\mathbf{x}_i, y_i) \perp (\mathbf{x}_j, y_j)$$ for $$i \neq j$$.

But I think we can get the same with just assuming conditional independence i.e $$y_i \mid \mathbf{x}_i \perp y_j \mid \mathbf{x}_j$$ for $$i \neq j$$. But, I do not think that this notation is valid, is the correct notation $$y_i \perp y_j \mid \mathbf{x}_j, \mathbf{x}_i$$. But then I do not think this correct as that statement only allows us to write:

$$p(y_i, y_j \mid \mathbf{x}_i, \mathbf{x}_j) = p(y_i \mid \mathbf{x}_i, \mathbf{x}_j) p(y_j \mid \mathbf{x}_i, \mathbf{x}_j)$$

But we really want:

$$p(y_i, y_j \mid \mathbf{x}_i, \mathbf{x}_j) = p(y_i \mid \mathbf{x}_i) p(y_j \mid \mathbf{x}_j)$$

In summary, my question basically is, what is the correct way to write the conditional independent statements for linear regression?

• You need all the responses to be uncorrelated with all the explanatory variables. You can find a great deal written about this here on CV: see this site search.
– whuber
Commented Jun 8, 2023 at 14:44
• Ok thank you, but I was wondering how you would write these independence statements mathematically. I was also wondering in PRML book what independence statement is used that allows them to write equation 3.10? And I am guessing that in a fixed design setting we can just say: assume responses are independent? Commented Jun 12, 2023 at 9:19
• @whuber I also was wondering if you had a recommended book for a rigorous description of linear regression. That provides the assumptions, and the description of inferential statements that can be made under the assumptions. Thanks again. Commented Jun 12, 2023 at 10:58