4
$\begingroup$

Could someone please clarify the part highlighted in red? Why the conditional density? I am having hard time understanding why the statement is about conditional density

I don't understand why saying that $e∼N(0,σ^2)$ implies that $Y∣X$ is Normal. I would like to have a proof .. not just words about the concept

thank youenter image description here

$\endgroup$
  • 3
    $\begingroup$ It's an attempt to apply the definition of "$e\sim N(0,\sigma^2)$". All that's needed is to plug $e = y-(x^\prime\beta+ \alpha)$ into the usual formula for the Normal density. Somehow $\alpha$ was omitted--but isn't it obvious where it belongs? $\endgroup$ – whuber Sep 18 '17 at 20:14
  • $\begingroup$ I don't understand why saying that $ e \sim N( 0, \sigma^2)$ implies that $Y \mid X $. I would like to have a proof .. not just words about the concept $\endgroup$ – Hard Core Sep 19 '17 at 11:12
  • $\begingroup$ @HardCore What do you mean "$e \sim N(0, \sigma)$ implies that Y|X"? Are you misreading the conditional operator to mean that $Y$ depends on $X$? $Y,X$ are (in this presentation) jointly observed RVs, so a conditional density exists. Since $e$ is independent of $X$, $Y|X$ is a sequence of constants plus a normal RV, making $Y|X$ normal. $\endgroup$ – AdamO Sep 19 '17 at 12:36
  • $\begingroup$ In the main post I highlighted a sentence in red. That sentence says the same thing I wrote. You are right !!! I was gonna write $ e \sim N(0,\sigma^2)$ implies that $Y \mid X$ is normal. Sorry $\endgroup$ – Hard Core Sep 19 '17 at 12:46
  • 3
    $\begingroup$ @HardCore But regression model conditions on the observed (i.e. known) X's. Moreover, pleas do not be rude. AdamO is trying to help and you are being rude to him. $\endgroup$ – Tim Sep 19 '17 at 13:24
2
$\begingroup$

Simple linear regression model (let's focus on single predictor case for simplicity) describes relationship of dependent variable $Y$ with independent variable $X$. It tells us what kind of value of $Y$ can we expect when $X=y$, i.e. it models conditional expectation

$$ E(Y|X) = \mu = \alpha + \beta X $$

where $Y|X \sim \mathcal{N}(\mu, \sigma^2)$, since it is $e + \mu$, where $e \sim \mathcal{N}(0, \sigma^2)$. It is conditional by definition, because we are interested in the relationship between the variables. If it weren't, we would simply ask "what is the expected value of $Y$?", and we wouldn't care about $X$. Finally, if $Y$ and $X$ were independent, then the model would simplify to

$$ E(Y|X) = \alpha + 0 \times X = \alpha $$

where $\alpha$ would be a single-value summary statistic that describes $Y$ and minimizes the squared error, i.e. the mean of $Y$.

$\endgroup$
  • $\begingroup$ I don't understand why saying that $e∼N(0,σ^2)$ implies that $Y∣X$ is Normal. I would like to have a proof .. not just words about the concept $\endgroup$ – Hard Core Sep 19 '17 at 12:19
  • $\begingroup$ @HardCore this comes from the properties of expectations $E(X+c) = E(X) +c$. If you add a constant to the random variable you shift it's mean by the constant. Same if you add random variables $E(X+Y) =E(X)+E(Y)$ and if you sum normally distributed random variables. You are simply shifting the mean from $0$ to $\mu$. $\endgroup$ – Tim Sep 19 '17 at 12:23
  • $\begingroup$ Yes of course I know that, but first you consider the uncoditional and then the conditional. And again why you all keep saying that $X$ is fixed? $X$ is a random variable that has a joint distribution with $Y$. If I want to study the effect say of the weight Y on the height X, when I draw a sample I observe jointly the 2 RVs $X$ and $Y$. They have a joint density $f(x,y)$. They are both RV. Why $X$ should be fixed in this context? $\endgroup$ – Hard Core Sep 19 '17 at 12:26
  • $\begingroup$ Tim I think we are talking about different things. Thank you very much for the aswer though. $\endgroup$ – Hard Core Sep 19 '17 at 12:31
  • $\begingroup$ @HardCore when using regression you don't estimate the joint distribution, you simplify this problem to estimating the conditional expectation by conditioning on the observed $X$'s. $\endgroup$ – Tim Sep 19 '17 at 12:36
0
$\begingroup$

In regression modeling, the $X$ (a matrix or vector of covariate values) is considered fixed or given by design. Whether by randomization assignment, or by sampling strategy even simple random sampling, this is often a reflection of how data are collected. If we think of $X$ as a constant, then we can speak simply of the density of the $Y$.

$Y$ then is a bunch of constants plus a normal term, making the $Y$s normally distributed with some mean and standard deviation. This is nice because we can use our observed variables ($Y$ and $X$) to do maximum likelihood and estimate $\beta$ even if we never observe $\epsilon$. If we knew $\epsilon$, regression would be simple arithmetic: subtract residual error, solve a system of equations.

$\endgroup$
  • $\begingroup$ I can't see how $X$ can be considered fixed as it is a random variable. In fact the conditional expectation $E(Y \mid X)$ itself is a random variable. When you take conditional expectation you are actually conditioning on the $\sigma-field$ generated by $X$, $\sigma(X)$. And you couldn't do it if X were fixed. Every time you draw a sample you observe a different value of X. Of course once the sample has been drawn X is fixed, but in that case everything is fixed. $\endgroup$ – Hard Core Sep 19 '17 at 12:22
  • $\begingroup$ @HardCore this is a basic tenant of probability theory: when you condition on something, it is no longer random. In a formula: $E[X|X] = X$. If I throw a coin, it is either heads or tails, the randomness comes about when I speak of repeatedly throwing the coin in the same fashion. $\endgroup$ – AdamO Sep 26 '17 at 19:58
0
$\begingroup$

After a while, I came back to this question that has not received an answer so far.

I was able to give a formal proof of the statement (finally). I will give one in the one dimensional case. The proof with $k$ regressors is simply an extension and does not affect the proof below.

Let's begin stating the hypotesis

  1. $Y = \beta X + e $

  2. $e \sim N(0, \sigma^2)$

  3. $X$ and $e$ are independent

We want to prove that $ f_{Y \mid X}(y \mid x)$ is $N(\beta x, \sigma^2)$

We are not making any assumption about the distribution of $X$, and I want to point that $X$ is a RANDOM unlike many people said that $X$ is considered FIXED.

Consequently $X$ has a probability distribution. That being said let's begin with the proof:

$f_{Y \mid X}(y \mid x) = \dfrac{f_{YX}(y,x)}{f_{X}(x)}$

$f_{YX}(y,x)=f_{YX}(\beta X + e,x) = f_{eX}(y- \beta X,x)$

the last equality is true because it is a simple bivariate transformation

The independence of $e$ and $X$ implies that

$f_{eX}(y- \beta X,x) = f_e(y- \beta X)f_X(x)$

Finally, since $e \sim N(0, \sigma^2)$

$f_{Y \mid X}(y \mid x) = f_e(y- \beta X) = \dfrac{1}{2 \sigma \sqrt{(\pi)}}exp{\bigg(\dfrac{1}{2} \Big(\dfrac{y -\beta x}{\sigma} \Big)^2 \bigg)} $

This conclude the proof since the last equality implies that.

$ Y \mid X \sim N(\beta x, \sigma^2)$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.