3
$\begingroup$

Question. If $Y$ is a sub-Gaussian random variable, must the regression residual $\varepsilon=Y-E(Y\mid X)$ be also sub-Gaussian? That is, whether there exists some constant $\sigma>0$ such that $E\big( e^{\lambda \varepsilon} \big) \leq e^{\lambda^2\sigma^2/2}$ for any $\lambda\in\mathbb R$.

In terms of the variance, it is well known that $\mathrm{var}(\varepsilon)\leq \mathrm{var}(Y)$. However, I can't find out a way to prove (or disprove) the sub-Gaussianity of $\varepsilon$ when $Y$ is sub-Gaussian. I wonder if $\varepsilon$ is always sub-Gaussian, by the sub-Gaussianity of $Y$, and maybe plus some additional requirements on $X$ (such as $X$ is also sub-Gaussian) or $E(Y\mid X)$ (such as $E(Y\mid X)$ is linear).

$\endgroup$
3
  • $\begingroup$ The question isn't well-defined, because $\varepsilon$ has a bivariate distribution. If $X$ is not subgaussian, then in what sense would you hope for $\varepsilon$ to be subgaussian? $\endgroup$
    – whuber
    Commented Nov 22, 2022 at 3:34
  • 1
    $\begingroup$ @whuber I have updated the question by providing the definition of the sub-Gaussianity of $\varepsilon$. Actually, although $\varepsilon$ is generated by a bivariate distribution, I suppose it can be treated as a random variable with a univariate distribution. $\endgroup$
    – Zhao Zhao
    Commented Nov 22, 2022 at 4:19
  • $\begingroup$ It turns out the statement is true, with a proof provided in math.stackexchange.com/questions/4581889/…. $\endgroup$
    – Zhao Zhao
    Commented Nov 22, 2022 at 11:58

1 Answer 1

0
$\begingroup$

If you're dealing with linear regression - the answer is yes, $\epsilon$ is indeed subG.

First and most obvious, $E[\epsilon]=E[Y]-E[E[Y|X]]=0$ by law of total expectation. Then, according to Vershynin, all you need to prove is the bound on the MGF of $\epsilon$.

Now, if you're modelling linear regression then $E[Y|X]=\hat{Y}\sim\mathcal{N}$, which means this is a subG variable as well and by Hoeffding's lemma we get that their sum is also subG.

This won't hold for Poisson regression, as the Poisson distribution is not subG.

Edit:

The thing with subG property is this: you need the distribution to fulfill that, from a point $\lambda>0$, the probability tail $P(|Y|>\lambda)$ is lighter than the Gaussian tail. This is usually the case when $Y$ is some variant of Gaussian or when it is bounded. If $Y$ is subG with variance proxy $\sigma^2$ then by definition of regression (classic, not Bayesian) we get that $\hat{Y}$ has subG distribution with the same proxy, hence the residual is subG with this proxy as well. The only condition here is that $E[Y|X]=E[Y]$ - that is, $\hat{Y}$ should be an unbiased estimator.

$\endgroup$
4
  • $\begingroup$ Thanks for your reply. In your Poisson regression example, $Y$ itself is not sub-Gaussian. But I wonder if it is possible $\varepsilon$ is not sub-Gaussian when $Y$ is sub-Gaussian. $\endgroup$
    – Zhao Zhao
    Commented Nov 22, 2022 at 8:34
  • $\begingroup$ see edit regarding tails and condition $\endgroup$
    – Spätzle
    Commented Nov 22, 2022 at 8:57
  • $\begingroup$ The statement "by definition of regression (classic, not Bayesian) we get that $\widehat{Y}$ has sub-Gaussian distribution with the same proxy" seems less obvious to me. Would you please provide a proof or a reference for this? $\endgroup$
    – Zhao Zhao
    Commented Nov 22, 2022 at 9:17
  • $\begingroup$ When constructing a GLM model, the most basic assumption is $\hat{Y}_i=E[Y_i|X_i]$. If, for example, we discuss linear regression then the general model is $y_i=x^T_i\beta+\epsilon_i$, where $\epsilon_i\sim\mathcal{N}(0,\sigma^2)$. It then follows that due to the unbiasedness nature of $\hat{y}_i$, we get $y_i-\hat{y}_i=e_i\sim\mathcal{N}(0,\sigma^2)$. $\endgroup$
    – Spätzle
    Commented Nov 22, 2022 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.