Maximum Likelihood Formulation for Linear Regression

Question

I have seen the following for maximum likelihood estimation (MLE) for linear regression in multiple sources, e.g. here:

$$ \mathcal{D} \equiv \{(x_1, y_1), ..., (x_n, y_n)\} $$

I do not understand how exactly we derive this:

$$ p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i | x_i, \theta) $$

I understand that we can write the product due to the assumtion of independent $y_i$. However, I do not understand why $x_i$ is suddenly on the right side. Shouldn't it be:

$$ p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i , x_i | \theta) $$

Answer 1

In ordinary least squares regression the goal is to model the condition expectation; $$ E[y_i|x_i] = x_i'\beta $$

$y_i$ and $x_i$ are referred to as the dependent and independent variables respectively because we are literally conditioning $y_i$ on $x_i$.

Ordinary least squares is equivalent to maximum likelihood where we assume; $$ y_i|x_i \stackrel{iid}{\sim} N(x_i'\beta,\sigma^2) $$ In this instance the $x_i$ are taken as fixed values (we are not calling $x_i$ a random variable and giving it a probability distribution) meaning that the "data", $\mathcal{D}$, is just the set of $y_i$'s $$\mathcal{D} \equiv \{y_1,..,y_n\}$$

So writing $$ p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i | x_i, \theta) $$

where $\theta \equiv \{\beta,\sigma\}$ is actually correct.

The likelihood $p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i , x_i | \theta)=\prod_{i=1}^n p_y(y_i | x_i, \theta)p_x(x_i|\theta)$ ,on the other hand, treats the $x_i$ as random variables which, although applicable in some settings, is not linear regression in the traditional sense.