# Do OLS residuals tell us anything about the distribution of the error term? [duplicate]

I'd like some intuition on a question that has long confused me. Suppose we have a data-generating process

$$y_i = x_i' \beta + \varepsilon_i$$

where $$\mathbb{E}[\varepsilon_i] = 0$$, $$\varepsilon_i \perp x_i$$, and $$\varepsilon_i$$ is drawn i.i.d. from a probability distribution $$U$$.

When we regress $$y$$ on $$x$$ in a finite sample, we will get a set of residuals $$\hat{\varepsilon}_i$$. My question is, do these residuals tell us anything about $$U$$?

It seems obvious that the greater the variance of $$U$$ (i.e. when the DGP is noisier), the greater we can expect the empirical variance of $$\hat{\varepsilon}_i$$ to be, so clearly they must be somewhat related. Moreover, if you indeed had $$y$$ and $$x$$ for the entire population, there would be no estimation error, so the empirical distribution of $$\hat{\varepsilon}_i$$ would match $$U$$ exactly. So, for finite samples, what can residuals tell us about the shape of the unknown distribution $$U$$?

## marked as duplicate by Ben, kjetil b halvorsen, Peter Flom♦ regression StackExchange.ready(function() { if (StackExchange.options.isMobile) return; $('.dupe-hammer-message-hover:not(.hover-bound)').each(function() { var$hover = $(this).addClass('hover-bound'),$msg = $hover.siblings('.dupe-hammer-message');$hover.hover( function() { $hover.showInfoMessage('', { messageElement:$msg.clone().show(), transient: false, position: { my: 'bottom left', at: 'top center', offsetTop: -7 }, dismissable: false, relativeToBody: true }); }, function() { StackExchange.helpers.removeMessages(); } ); }); }); Apr 24 at 11:11

• The answer depends on how you perform the regression. Are you thinking of ordinary least squares regression? Note that even when you have the entire population, the regression estimates and residuals might still be wrong: you also need the right model and you need a suitable regression technique for fitting it. – whuber Apr 24 at 2:39
• If you fit the 'true' model, your residuals are linear combinations of errors $r = (I-H)\epsilon$ (fitting by least squares and assuming it has an intercept). Typically (where leverages are small) residuals are highly correlated to errors -- but each residual is in general a different combination (each with its different distribution); if you combine those residuals you typically end up with a kind of "smeared" version of the original. It's easy to investigate behaviour via simulation for various hat-matrices ($H=X(X^\top X)^{-1}X^\top$) and error-distributions. – Glen_b Apr 24 at 2:57
• @whuber: yes, I should have mentioned this in the body of my question but from the title I did mean OLS. Further, as implied from my use of the term "data-generating process," the model is correctly specified. – Kenneth Apr 24 at 20:22
• @Glen_b: you're on the right track and I've done a bit of simulating with the $H$ from OLS that you provided, but actually got very little correlation between residuals and errors! – Kenneth Apr 24 at 20:25

a <- rbinom(100, 1, .5)
ols <- lm(holder$$a ~ holder$$X1.length.a.)