I'm investigating interplay between two variables ($x_1$ and $x_2$). There is a great deal of linear correlation between these variables with $r>0.9$. From the nature of the problem I cannot say anything about the causation (whether $x_1$ causes $x_2$ or the other way around). I would like to study deviations from the regression line, in order to detect outliers. In order to do this I can either build a linear regression of $x_1$ as a function of $x_2$, or the other way around. Can my choice of variable order influence my results?
5
-
$\begingroup$ In the search for outliers, you should first regress your dependent variable against both $x_{1}$ and $x_{2}$ and look for outliers. $\endgroup$– schenectadyCommented May 15, 2011 at 17:09
-
$\begingroup$ Is finding outliers the pupose of your investigation? If so, then you should first regress your dependent variable against both $x_{1}$ and $x_{2}$ and then perform outlier tests. If finding possible causation, then you should consider performing a designed experiment. If the purpose of your experiment is to find a relationship between your two independent variables, looking at a happenstance of collected data will not do the trick. $\endgroup$– schenectadyCommented May 15, 2011 at 17:15
-
$\begingroup$ It isn't clear to me what you mean by outliers. If there are outliers in your data then they will affect the calculation of the regression line. Why are you looking for outliers in both $x_1$ and $x_2$ simultaneously? $\endgroup$– DQdlMCommented May 15, 2011 at 19:28
-
$\begingroup$ See stats.stackexchange.com/questions/22718/… $\endgroup$– kjetil b halvorsen ♦Commented Sep 4, 2020 at 3:38
-
$\begingroup$ Please clarify the roles of these "variables": are they both predictors of a third variable? If not, then the links suggested in earlier comments identify a duplicate thread. $\endgroup$– whuber ♦Commented Sep 4, 2020 at 14:09
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
It surely can (actually, it even matters with regard to the assumptions on your data - you only make assumptions about the distribution of the outcome given the covariate). In this light, you might look up a term like "inverse prediction variance". Either way, linear regression says nothing about causation! At best, you can say something about causation through careful design.
answered May 15, 2011 at 15:53
$\begingroup$
$\endgroup$
To make the case symmetrical, one may regress the difference between the two variables ($\Delta x$) vs their average value.
answered May 15, 2011 at 16:53
$\begingroup$
$\endgroup$
Standard regression minimizes the vertical distance between the points and the line, so switching the 2 variables will now minimize the horizontal distance (given the same scatterplot). Another option (which goes by several names) is to minimize the perpendicular distance, this can be done using principle components.
Here is some R code that shows the differences:
library(MASS)
tmp <- mvrnorm(100, c(0,0), rbind( c(1,.9),c(.9,1)) )
plot(tmp, asp=1)
fit1 <- lm(tmp[,1] ~ tmp[,2]) # horizontal residuals
segments( tmp[,1], tmp[,2], fitted(fit1),tmp[,2], col='blue' )
o <- order(tmp[,2])
lines( fitted(fit1)[o], tmp[o,2], col='blue' )
fit2 <- lm(tmp[,2] ~ tmp[,1]) # vertical residuals
segments( tmp[,1], tmp[,2], tmp[,1], fitted(fit2), col='green' )
o <- order(tmp[,1])
lines( tmp[o,1], fitted(fit2)[o], col='green' )
fit3 <- prcomp(tmp)
b <- -fit3$rotation[1,2]/fit3$rotation[2,2]
a <- fit3$center[2] - b*fit3$center[1]
abline(a,b, col='red')
segments(tmp[,1], tmp[,2], tmp[,1]-fit3$x[,2]*fit3$rotation[1,2], tmp[,2]-fit3$x[,2]*fit3$rotation[2,2], col='red')
legend('bottomright', legend=c('Horizontal','Vertical','Perpendicular'), lty=1, col=c('blue','green','red'))
To look for outliers you can just plot the results of the principle components analysis.
You may also want to look at:
Bland and Altman (1986), Statistical Methods for Assessing Agreement Between Two Methods of CLinical Measurement. Lancet, pp 307-310
answered May 16, 2011 at 15:13
$\begingroup$
$\endgroup$
Your x1 and x2 variables are collinear. In the presence of multicollinearity, your parameter estimates are still unbiased, but their variance is large, i.e., your inference on the significance of the parameter estimates is not valid, and your prediction will have large confidence intervals.
Interpretation of the parameter estimates is also difficult. In the linear regression framework, the parameter estimate on x1 is the change in Y for a unit change in x1 given every other exogeneous variable in the model is held constant. In your case, x1 and x2 are highly correlated, and you cannot hold x2 constant when x1 is changing.
