In an interview question I was asked what would be the effect of adding a Gaussian noise to the features $\mathbf X\in\mathbb R^{n\times k}$ over the confidence interval of the parameters $\mathbf\beta\in\mathbb R^k$.
I could not answer, so afterwards I went checking around how these confidence intervals look like, and found that the parameters have associated a $t$-statistics $(\beta_j-\beta_{0,j})/\sigma_j$, where $\sigma_j$ is the standard error of the $j$-th parameter:
$$ \sigma_j=\sqrt{\frac{\hat\sigma^2}{\Sigma_{jj}}},$$
where $\hat\sigma^2$ is the estimator of the variance of the residuals and $\mathbf\Sigma=\mathbf X^T\mathbf X$ is the sum-of-squares matrix of the features.
This suggests me that the confidence interval improves increasing the variance of the features, that is honesty counterintuitive, and that therefore adding Gaussian noise should also improve it. This is even more counterintuitive: shuffling the deck improves predictions about it? How would you explain this?
