6
$\begingroup$

I have seen the following for maximum likelihood estimation (MLE) for linear regression in multiple sources, e.g. here:

$$ \mathcal{D} \equiv \{(x_1, y_1), ..., (x_n, y_n)\} $$

I do not understand how exactly we derive this:

$$ p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i | x_i, \theta) $$

I understand that we can write the product due to the assumtion of independent $y_i$. However, I do not understand why $x_i$ is suddenly on the right side. Shouldn't it be:

$$ p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i , x_i | \theta) $$

$\endgroup$
9
  • $\begingroup$ I think it depends on what you are trying to model. In a linear regression you are trying to learn about $y$ from $x$ and thus you need the likelihood to be a function of $y$. In the case of linear regression you are treating $x$ as a quantity that helps you explain the thing you want to learn about, $y$, which is why you treat it as being conditioned upon as if it is fixed and known. $\endgroup$ Feb 11, 2016 at 7:32
  • $\begingroup$ I understand that we model $p(y|x,\theta)$, however I don't understand why this is equal to $p(\mathcal{D}|\theta)$ $\endgroup$ Feb 11, 2016 at 7:42
  • $\begingroup$ Abuse of notation maybe. $\endgroup$ Feb 11, 2016 at 7:44
  • 1
    $\begingroup$ @DaeyoungLim when someone is asking a question that basically boils down to "hey, I don't understand this notation", the answer of "You don't need to pay attention to the notation" makes no sense. $\endgroup$ Feb 11, 2016 at 8:11
  • 1
    $\begingroup$ @RustyStatistician Touché. I think you're right. $\endgroup$ Feb 11, 2016 at 8:24

1 Answer 1

10
$\begingroup$

In ordinary least squares regression the goal is to model the condition expectation; $$ E[y_i|x_i] = x_i'\beta $$

$y_i$ and $x_i$ are referred to as the dependent and independent variables respectively because we are literally conditioning $y_i$ on $x_i$.

Ordinary least squares is equivalent to maximum likelihood where we assume; $$ y_i|x_i \stackrel{iid}{\sim} N(x_i'\beta,\sigma^2) $$ In this instance the $x_i$ are taken as fixed values (we are not calling $x_i$ a random variable and giving it a probability distribution) meaning that the "data", $\mathcal{D}$, is just the set of $y_i$'s $$\mathcal{D} \equiv \{y_1,..,y_n\}$$

So writing $$ p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i | x_i, \theta) $$

where $\theta \equiv \{\beta,\sigma\}$ is actually correct.

The likelihood $p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i , x_i | \theta)=\prod_{i=1}^n p_y(y_i | x_i, \theta)p_x(x_i|\theta)$ ,on the other hand, treats the $x_i$ as random variables which, although applicable in some settings, is not linear regression in the traditional sense.

$\endgroup$
5
  • $\begingroup$ Thanks, that makes sense. Would it be also ok if we still define $\mathcal{D} \equiv \{ (y_1, x_1), ...., (y_n, x_n) \}$, as this is what we actually observe, but further make clear, that $y_i|x_i \stackrel{iid}{\sim} N(x_i'\beta,\sigma^2)$. So that the $y_i$ in $\mathcal{D}$ are samples from a random variable, with associated values $x_i$? $\endgroup$ Feb 11, 2016 at 8:34
  • 1
    $\begingroup$ Technically in this context the answer is no (although this is sort of minutiae). If we define $\mathcal{D} \equiv \{ (y_1, x_1), ...., (y_n, x_n) \}$ then $P(\mathcal{D} | \theta)$ is a bivariate distrbution and we are required to assign a marginal distribution to $x_i$, $p_x(x_i | \theta)$, which ordinary linear regression is not concerned with (see the last paragraph in the answer). You could certainty define a likelihood this way (time series and state-space models are a couple great examples) but it is not ordinary least squares regression. $\endgroup$ Feb 11, 2016 at 8:50
  • $\begingroup$ Thanks a lot! One last question, would it be possible to write $\mathcal{D} \equiv \{ (y_1 | x_1), ...., (y_n | x_n) \}$? $\endgroup$ Feb 11, 2016 at 9:10
  • $\begingroup$ It's interesting that you thought about it that way. I think that notation would technically be acceptable, although it will probably confuse most people. $\endgroup$ Feb 11, 2016 at 9:22
  • $\begingroup$ You could say that linear regression implies that the given inputs is exactly the distribution of $X$, i.e. $p(x_i) = 1_{x_i \in \mathcal{D}}$ $\endgroup$ Oct 16, 2020 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.