I have seen the following for maximum likelihood estimation (MLE) for linear regression in multiple sources, e.g. here:

$$ \mathcal{D} \equiv \{(x_1, y_1), ..., (x_n, y_n)\} $$

I do not understand how exactly we derive this:

$$ p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i | x_i, \theta) $$

I understand that we can write the product due to the assumtion of independent $y_i$. However, I do not understand why $x_i$ is suddenly on the right side. Shouldn't it be:

$$ p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i , x_i | \theta) $$

  • $\begingroup$ I think it depends on what you are trying to model. In a linear regression you are trying to learn about $y$ from $x$ and thus you need the likelihood to be a function of $y$. In the case of linear regression you are treating $x$ as a quantity that helps you explain the thing you want to learn about, $y$, which is why you treat it as being conditioned upon as if it is fixed and known. $\endgroup$ – RustyStatistician Feb 11 '16 at 7:32
  • $\begingroup$ I understand that we model $p(y|x,\theta)$, however I don't understand why this is equal to $p(\mathcal{D}|\theta)$ $\endgroup$ – user3429986 Feb 11 '16 at 7:42
  • $\begingroup$ Abuse of notation maybe. $\endgroup$ – RustyStatistician Feb 11 '16 at 7:44
  • 1
    $\begingroup$ @DaeyoungLim when someone is asking a question that basically boils down to "hey, I don't understand this notation", the answer of "You don't need to pay attention to the notation" makes no sense. $\endgroup$ – RustyStatistician Feb 11 '16 at 8:11
  • 1
    $\begingroup$ @RustyStatistician Touché. I think you're right. $\endgroup$ – Daeyoung Lim Feb 11 '16 at 8:24

In ordinary least squares regression the goal is to model the condition expectation; $$ E[y_i|x_i] = x_i'\beta $$

$y_i$ and $x_i$ are referred to as the dependent and independent variables respectively because we are literally conditioning $y_i$ on $x_i$.

Ordinary least squares is equivalent to maximum likelihood where we assume; $$ y_i|x_i \stackrel{iid}{\sim} N(x_i'\beta,\sigma^2) $$ In this instance the $x_i$ are taken as fixed values (we are not calling $x_i$ a random variable and giving it a probability distribution) meaning that the "data", $\mathcal{D}$, is just the set of $y_i$'s $$\mathcal{D} \equiv \{y_1,..,y_n\}$$

So writing $$ p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i | x_i, \theta) $$

where $\theta \equiv \{\beta,\sigma\}$ is actually correct.

The likelihood $p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i , x_i | \theta)=\prod_{i=1}^n p_y(y_i | x_i, \theta)p_x(x_i|\theta)$ ,on the other hand, treats the $x_i$ as random variables which, although applicable in some settings, is not linear regression in the traditional sense.

  • $\begingroup$ Thanks, that makes sense. Would it be also ok if we still define $\mathcal{D} \equiv \{ (y_1, x_1), ...., (y_n, x_n) \}$, as this is what we actually observe, but further make clear, that $y_i|x_i \stackrel{iid}{\sim} N(x_i'\beta,\sigma^2)$. So that the $y_i$ in $\mathcal{D}$ are samples from a random variable, with associated values $x_i$? $\endgroup$ – user3429986 Feb 11 '16 at 8:34
  • 1
    $\begingroup$ Technically in this context the answer is no (although this is sort of minutiae). If we define $\mathcal{D} \equiv \{ (y_1, x_1), ...., (y_n, x_n) \}$ then $P(\mathcal{D} | \theta)$ is a bivariate distrbution and we are required to assign a marginal distribution to $x_i$, $p_x(x_i | \theta)$, which ordinary linear regression is not concerned with (see the last paragraph in the answer). You could certainty define a likelihood this way (time series and state-space models are a couple great examples) but it is not ordinary least squares regression. $\endgroup$ – Zachary Blumenfeld Feb 11 '16 at 8:50
  • $\begingroup$ Thanks a lot! One last question, would it be possible to write $\mathcal{D} \equiv \{ (y_1 | x_1), ...., (y_n | x_n) \}$? $\endgroup$ – user3429986 Feb 11 '16 at 9:10
  • $\begingroup$ It's interesting that you thought about it that way. I think that notation would technically be acceptable, although it will probably confuse most people. $\endgroup$ – Zachary Blumenfeld Feb 11 '16 at 9:22
  • $\begingroup$ You could say that linear regression implies that the given inputs is exactly the distribution of $X$, i.e. $p(x_i) = 1_{x_i \in \mathcal{D}}$ $\endgroup$ – Mr Tsjolder Oct 16 '20 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.