I know that in linear regression, the marginal distribution of $y$ is usually not of interest. But, just out of curiosity, what is the marginal distribution of the response variable, in the case of the multiple linear regression model? I can think of two cases:

The predictors are not random variables

In this case (by far the most common), we know that

$$ p(y|\mathbf{x}) = \mathcal{N}(\beta_0+\boldsymbol{\beta}^T\cdot\mathbf{x},\sigma^2)$$

Since $\mathbf{x}$ is not a random vectors, it doesn't make sense to think of the joint distribution of $y$ and $\mathbf{x}$. I think in this case $y$ must be considered a stochastic process, i.e., a function from $\mathbb{R}^d\times\Omega\to\mathbb{R}$ ($d$ is the number of predictors, of course). For each given $\mathbf{x}$, $y|\mathbf{x}$ is a random variable and it makes sense to ask about its distribution (which is just a normal distribution as shown above). However, it doesn't make sense to think of "marginalizing" this distribution with respect to $\mathbf{x}$, since $\mathbf{x}$ is not random.

The predictors are random variables

As before, $$ p(y|\mathbf{x}) = \mathcal{N}(\beta_0+\boldsymbol{\beta}^T\cdot\mathbf{x},\sigma^2)$$

but now this is a "real" conditional distribution, so we can marginalize. In practice

$$ p(y) =\int p(y,\mathbf{x})\text{d}\mathbf{x}=\int p(y|\mathbf{x})p(\mathbf{x})\text{d}\mathbf{x}$$

Now it all depends on the joint distribution of the $d$ predictors. Depending on $p(\mathbf{x})$, the marginal distribution of $y$ could be basically anything.

What do you think? Is my reasoning correct, or am I missing something?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.