Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have seen the following for maximum likelihood estimation (MLE) for linear regression in multiple sources, e.g. here:

$$ \mathcal{D} \equiv \{(x_1, y_1), ..., (x_n, y_n)\} $$

I do not understand how exactly we derive this:

$$ p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i | x_i, \theta) $$

I understand that we can write the product due to the assumtion of independent $y_i$. However, I do not understand why $x_i$ is suddenly on the right side. Shouldn't it be:

$$ p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i , x_i | \theta) $$

share|improve this question
    
I think it depends on what you are trying to model. In a linear regression you are trying to learn about $y$ from $x$ and thus you need the likelihood to be a function of $y$. In the case of linear regression you are treating $x$ as a quantity that helps you explain the thing you want to learn about, $y$, which is why you treat it as being conditioned upon as if it is fixed and known. – RustyStatistician Feb 11 at 7:32
    
I understand that we model $p(y|x,\theta)$, however I don't understand why this is equal to $p(\mathcal{D}|\theta)$ – user3429986 Feb 11 at 7:42
    
Abuse of notation maybe. – RustyStatistician Feb 11 at 7:44
1  
@DaeyoungLim when someone is asking a question that basically boils down to "hey, I don't understand this notation", the answer of "You don't need to pay attention to the notation" makes no sense. – RustyStatistician Feb 11 at 8:11
1  
@RustyStatistician Touché. I think you're right. – Daeyoung Lim Feb 11 at 8:24
up vote 9 down vote accepted

In ordinary least squares regression the goal is to model the condition expectation; $$ E[y_i|x_i] = x_i'\beta $$

$y_i$ and $x_i$ are referred to as the dependent and independent variables respectively because we are literally conditioning $y_i$ on $x_i$.

Ordinary least squares is equivalent to maximum likelihood where we assume; $$ y_i|x_i \stackrel{iid}{\sim} N(x_i'\beta,\sigma^2) $$ In this instance the $x_i$ are taken as fixed values (we are not calling $x_i$ a random variable and giving it a probability distribution) meaning that the "data", $\mathcal{D}$, is just the set of $y_i$'s $$\mathcal{D} \equiv \{y_1,..,y_n\}$$

So writing $$ p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i | x_i, \theta) $$

where $\theta \equiv \{\beta,\sigma\}$ is actually correct.

The likelihood $p(\mathcal{D} | \theta) = \prod_{i=1}^n p(y_i , x_i | \theta)=\prod_{i=1}^n p_y(y_i | x_i, \theta)p_x(x_i|\theta)$ ,on the other hand, treats the $x_i$ as random variables which, although applicable in some settings, is not linear regression in the traditional sense.

share|improve this answer
    
Thanks, that makes sense. Would it be also ok if we still define $\mathcal{D} \equiv \{ (y_1, x_1), ...., (y_n, x_n) \}$, as this is what we actually observe, but further make clear, that $y_i|x_i \stackrel{iid}{\sim} N(x_i'\beta,\sigma^2)$. So that the $y_i$ in $\mathcal{D}$ are samples from a random variable, with associated values $x_i$? – user3429986 Feb 11 at 8:34
1  
Technically in this context the answer is no (although this is sort of minutiae). If we define $\mathcal{D} \equiv \{ (y_1, x_1), ...., (y_n, x_n) \}$ then $P(\mathcal{D} | \theta)$ is a bivariate distrbution and we are required to assign a marginal distribution to $x_i$, $p_x(x_i | \theta)$, which ordinary linear regression is not concerned with (see the last paragraph in the answer). You could certainty define a likelihood this way (time series and state-space models are a couple great examples) but it is not ordinary least squares regression. – Zachary Blumenfeld Feb 11 at 8:50
    
Thanks a lot! One last question, would it be possible to write $\mathcal{D} \equiv \{ (y_1 | x_1), ...., (y_n | x_n) \}$? – user3429986 Feb 11 at 9:10
    
It's interesting that you thought about it that way. I think that notation would technically be acceptable, although it will probably confuse most people. – Zachary Blumenfeld Feb 11 at 9:22

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.