# Maximum Likelihood Formulation for Linear Regression

I have seen the following for maximum likelihood estimation (MLE) for linear regression in multiple sources, e.g. here:

$$\mathcal{D} \equiv \{(x_1, y_1), ..., (x_n, y_n)\}$$

I do not understand how exactly we derive this:

$$p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i | x_i, \theta)$$

I understand that we can write the product due to the assumtion of independent $y_i$. However, I do not understand why $x_i$ is suddenly on the right side. Shouldn't it be:

$$p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i , x_i | \theta)$$

• I think it depends on what you are trying to model. In a linear regression you are trying to learn about $y$ from $x$ and thus you need the likelihood to be a function of $y$. In the case of linear regression you are treating $x$ as a quantity that helps you explain the thing you want to learn about, $y$, which is why you treat it as being conditioned upon as if it is fixed and known. Feb 11, 2016 at 7:32
• I understand that we model $p(y|x,\theta)$, however I don't understand why this is equal to $p(\mathcal{D}|\theta)$ Feb 11, 2016 at 7:42
• Abuse of notation maybe. Feb 11, 2016 at 7:44
• @DaeyoungLim when someone is asking a question that basically boils down to "hey, I don't understand this notation", the answer of "You don't need to pay attention to the notation" makes no sense. Feb 11, 2016 at 8:11
• @RustyStatistician Touché. I think you're right. Feb 11, 2016 at 8:24

In ordinary least squares regression the goal is to model the condition expectation; $$E[y_i|x_i] = x_i'\beta$$

$y_i$ and $x_i$ are referred to as the dependent and independent variables respectively because we are literally conditioning $y_i$ on $x_i$.

Ordinary least squares is equivalent to maximum likelihood where we assume; $$y_i|x_i \stackrel{iid}{\sim} N(x_i'\beta,\sigma^2)$$ In this instance the $x_i$ are taken as fixed values (we are not calling $x_i$ a random variable and giving it a probability distribution) meaning that the "data", $\mathcal{D}$, is just the set of $y_i$'s $$\mathcal{D} \equiv \{y_1,..,y_n\}$$

So writing $$p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i | x_i, \theta)$$

where $\theta \equiv \{\beta,\sigma\}$ is actually correct.

The likelihood $p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i , x_i | \theta)=\prod_{i=1}^n p_y(y_i | x_i, \theta)p_x(x_i|\theta)$ ,on the other hand, treats the $x_i$ as random variables which, although applicable in some settings, is not linear regression in the traditional sense.

• Thanks, that makes sense. Would it be also ok if we still define $\mathcal{D} \equiv \{ (y_1, x_1), ...., (y_n, x_n) \}$, as this is what we actually observe, but further make clear, that $y_i|x_i \stackrel{iid}{\sim} N(x_i'\beta,\sigma^2)$. So that the $y_i$ in $\mathcal{D}$ are samples from a random variable, with associated values $x_i$? Feb 11, 2016 at 8:34
• Technically in this context the answer is no (although this is sort of minutiae). If we define $\mathcal{D} \equiv \{ (y_1, x_1), ...., (y_n, x_n) \}$ then $P(\mathcal{D} | \theta)$ is a bivariate distrbution and we are required to assign a marginal distribution to $x_i$, $p_x(x_i | \theta)$, which ordinary linear regression is not concerned with (see the last paragraph in the answer). You could certainty define a likelihood this way (time series and state-space models are a couple great examples) but it is not ordinary least squares regression. Feb 11, 2016 at 8:50
• Thanks a lot! One last question, would it be possible to write $\mathcal{D} \equiv \{ (y_1 | x_1), ...., (y_n | x_n) \}$? Feb 11, 2016 at 9:10
• It's interesting that you thought about it that way. I think that notation would technically be acceptable, although it will probably confuse most people. Feb 11, 2016 at 9:22
• You could say that linear regression implies that the given inputs is exactly the distribution of $X$, i.e. $p(x_i) = 1_{x_i \in \mathcal{D}}$ Oct 16, 2020 at 14:45