2
$\begingroup$

Suppose we have n data observations $\left\{y_i, \underline{x_i}\right\}_{i=1}^n$. We can concatenate the $x_i$ into $X$.

We have $y_i=h^TX + \epsilon_i$.

I understand that, since we have observed it, the matrix $X$ becomes "non-random". $\epsilon_i$ is random noise. However, why do we still say that $y_i$ is random, conditioned on $X$? We have observed $y_i$ as well!

$\endgroup$
7
  • $\begingroup$ That is because $y_i$ is still random because conditioning on $X$ it is still a function of the random variable $e_i$ which does not depend on $X$. $\endgroup$ Commented Jan 18, 2017 at 18:48
  • 1
    $\begingroup$ But $X$ was random before observation. Why does the randomness of $y_i$ remain even after observation? We don't know the value of $e_i$, but we do know it for $y_i$. Why doesn't knowing the value make it non-random? $\endgroup$
    – learning
    Commented Jan 18, 2017 at 18:53
  • $\begingroup$ Conditioning on $X$ fixes it to be a constant vector. But $y_i$ is still a constant + $e_i$, $\endgroup$ Commented Jan 18, 2017 at 18:57
  • 1
    $\begingroup$ Could you explain what the $y_i$ value we are getting is? I think I am confused because it seems like $y_i$ itself is a constant after observation. It makes sense mathematically, but I can't figure out the intuition. I mean, we make observations on both $X$ and $y_i$, so it just confuses me. $\endgroup$
    – learning
    Commented Jan 18, 2017 at 19:00
  • 1
    $\begingroup$ I understand that observing $X$ makes it a constant. I was just wondering why observing $y_i$ (as we do for linear regression) as well doesn't mean that $y_i$ is now a constant. I don't know if that makes any sense. I think now it's just how we model it, with it being a realisation of the model (whereas $X$ is just from the underlying joint density, so we can condition it off). So $y=f(X,\epsilon)$ and hence still random. Maybe that's exactly what you are saying. $\endgroup$
    – learning
    Commented Jan 18, 2017 at 19:13

1 Answer 1

1
$\begingroup$

The fact that we have observed $X$ has nothing to do with it being non-random. It seems that it may be not clear for you what is meant in statistics by random variables.

Outcome of a flip of coin may be thought as a random variable. This does not mean that we don't know what was the outcome of the coin toss, but that the outcomes will vary from toss to toss, the behavior of the coin is not deterministic. Even more, in terms of physics, behavior of a coin is deterministic, but nevertheless, there is so many factors that determine each particular toss, that we think of it as of random variable. Other things can be considered as random variables as well, for example human age. There is nothing "random" in human ages, but if you will randomly approach people on the street and ask them for their ages, then there won't be any deterministic pattern in the ages (unless your sample is biased) and so, you could consider age in such experiment as random variable. Moreover, if you adopt broader, Bayesian definition of probability, then the notion of random variables can be extended even to events that are not obviously "random". To make it even more awkward, you can think of deterministic events (e.g. constant) as of random variables by thinking of them in terms of degenerate distribution. As you can see, this has literally nothing to do with the fact that you have observed something or not.

As about regression equation, the formula

$$ y_i = \beta_0 + \beta_1 x_{1i} + \dots + \beta_k x_{ki} + \varepsilon_i $$

may be written in terms of probabilistic model behind it as

$$ \mu_i = \beta_0 + \beta_1 x_{1i} + \dots + \beta_k x_{ki} \\ y_i \sim \mathcal{N}(\mu_i, \sigma^2) $$

so $Y$ is a random variable that follows normal distribution and it's mean is a function of $X$ and $\beta$. Moreover, also $X$ may be considered as random variable and, if using Bayesian approach, also $\beta$'s would be considered as random variables. What we mean by this is that we assume that such variables may take different values and the values have probabilities assigned to them, so we can describe them in terms of their probability distributions. It's about describing data in terms of probabilistic models.

If you were rather interested in what we mean by holding $X$ fixed, you may want to check Linear regression and interpretation of random variables and What is the difference between variable and random variable? threads.

$\endgroup$
6
  • $\begingroup$ Thanks for the very detailed reply! I know that X isn't non-random, more that it is a realisation of a random variable. I just didn't know what exactly to call it. "Fixed" is definitely a much better choice. Thanks for the resources! $\endgroup$
    – learning
    Commented Jan 18, 2017 at 22:49
  • $\begingroup$ Actually, is that right? $\endgroup$
    – learning
    Commented Jan 20, 2017 at 2:57
  • $\begingroup$ @radm94 is what right? $\endgroup$
    – Tim
    Commented Jan 21, 2017 at 8:01
  • $\begingroup$ I meant if I can indeed think of an observation of X as a realisation of the RV X. Then we can condition on an observation? $\endgroup$
    – learning
    Commented Jan 21, 2017 at 21:21
  • $\begingroup$ @radm94 you can condition only on random variables. Standard regression model assumes that X is fixed and mean of r.v. Y is a function of it. $\endgroup$
    – Tim
    Commented Jan 21, 2017 at 22:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.