I would like to demonstrate mathematically a thing, but I struggle to do this since I'm not a mathematician and I only have some high school remains of maths - I'm only an evolutionary biologist with curiosity for maths and stats.
Here's the pitch.
We have a dataset with three variables, with a fixed number n of individuals that are all scored for each variable. Let's name those variables x, y, and z. This dataset is, here, a sample of the real population. We're gonna do linear regressions (for now, let's stick to OLS to keep it simple) and the main interest of the study is to look at the residuals arising from these regressions.
First, we do a linear regression between y and x. Second, we do a linear regression between z and y. Third, we do a linear regression between z and x.
What I would like to demonstrate would be that doing the third regression is nonsense because all the variation this relationship could yield has been already "delivered" by the two previous ones, or, in other words, that the residuals of the z~x regression can be predicted using the residuals of the y~x and of z~y regressions. That's why I titled this post with "degrees of freedom", which are the number of really independent variables - here it's about the number of really independent regressions. To me, it's an intuitive thing, but I would need a "crystal clear" demonstration.
Here's what I've done so far.
First, in a perfect fit case, yes the z~x relationship can be predicted from the y~x and z~y relationships. If we have y = a.x+b, z = a'.y+b', and z = alpha.x+beta, then just by replacing the terms we've got z = a.a'.x+a'.b+b' and therefore alpha = a.a' and beta = a'.b+b.
Second, I considered a perfect fit with "incompressible residuals", which would be a case where the "perfect relationship" contains residuals (I guess that's not a very math thing, but at least it's a real biological thing). Adding residuals it then becomes a bit more complicated but still understandable; by considering r, r', and rho as the respective residuals of each regression (x~y, z~y, z~x), we finally have z = a'.a.x + a'.b + b + a'.r + r' then alpha = a'.a and beta+rho = a'.b + b + a'.r + r'.
What I'm struggling with is to, third, put and demonstrate this into a practical context, with empirical/simulated datas (I'm using R to do that), because empirical regressions do not estimate the real values of the regression parameters (because means of x/y/z are estimates and not real means). Knowing the true values of these parameters, and therefore the error of the estimation, it becomes OK. What I would like would be to either demonstrate how to retrieve "practical" residuals of the z~x regression using the residuals of the y~x and z~y regressions or to demonstrate that the gap between the observed and predicted residuals of the z~x regression is only due to the fact that we're using estimates of the regression parameters (and, ultimately, of the means of x, y, and z) and not "true" values.
Do not hesitate to ask for things if my post isn't fully clear. Many thanks!



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.