Return to Answer

I agree that ridge regression is arguably better, because it allows you to use the variables you had originally intended and is likely to yield betas that are very close to their true values (although they will be biased--see here or here for more information). Nonetheless, I think is also has two potential downsides: It is more complicated (requiring more statistical sophistication), and the resulting model is more difficult to interpret, in my opinion.

I agree that ridge regression is arguably better, because it allows you to use the variables you had originally intended and is likely to yield betas that are very close to their true values (although they will be biased--see here or here for more information). Nonetheless, I think is also has two potential downsides: It is more complicated (requiring more statistical sophistication), and the resulting model is more difficult to interpret, in my opinion.

First, (edited) in the real-world cases that I'm familiar with (data that I've worked with or read about) if one of a pair of very highly correlated variables had been thrown out, not much information would have been lost. (Be aware that this does not necessarily mean it's a good thing to do, I only mention this because it's often worth bearing in mind as a kind of baseline when you are faced with this situation.) This is likely due to the pattern of causal relationships that created the correlations between the variables in question and how they were related to the response variable. However, @whuber points out that it is completely possible for that not to be the case. I have no idea how often that kind of situation occurs versus the kinds that I'm more familiar with, but it's also worth bearing in mind.

Another thought is that you can also combine the variables. This is done by standardizing both (i.e., turning them into z-scores), averaging them, and then fitting your model with only the composite variable. This would be a good approach if you believe they are two different measures of one underlying construct. If you think one might be a better measure than the other, you could do a weighted average (perhaps using the reciprocals of the measurement errors).

As that implies, I suggest you think about what lies behind your correlated variables. That is, you need a theory of why they're so highly correlated to do the best job of picking which option to use. For example, @Macro's suggestion (which I've also used) assumes that you are primarily interested in $X$ and wonder about the additional contribution of $Z$ after having accounted for $X$'s contribution. Thus, thinking in this way will help you decide which (i.e., $x_1$ or $x_2$) should be treated as $X$ and which $Z$. (Edited) In line with what I said above, it's worth bearing in mind that the empirical / statistical properties of your model (e.g., goodness of fit) are likely to be quite similar whichever variable you choose for $X$ under many plausible situations (albeit perhaps not all). Thus, you want to use theoretical insight to inform your choice. For the sake of clarity, consider the following simulation:

library(MASS)
set.seed(8)
X = mvrnorm(n=200, mu=c(0,0), Sigma=rbind(c(1, .9),
                                          c(.9, 1)))
cor(X[,1], X[,2])     # [1] 0.9163053
y = 2 + 3*X[,1] + rnorm(200)

trueModel  = lm(y~X[,1])
wrongModel = lm(y~X[,2])

summary(trueModel)
...
Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  1.88429    0.07299   25.82   <2e-16 ***
X[, 1]       3.08083    0.06846   45.00   <2e-16 ***

Residual standard error: 1.03 on 198 degrees of freedom
Multiple R-squared: 0.9109, Adjusted R-squared: 0.9105 
F-statistic:  2025 on 1 and 198 DF,  p-value: < 2.2e-16 

summary(wrongModel)
...
Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   1.7586     0.1223   14.38   <2e-16 ***
X[, 2]        2.7916     0.1148   24.32   <2e-16 ***

Residual standard error: 1.729 on 198 degrees of freedom
Multiple R-squared: 0.7492, Adjusted R-squared: 0.7479 
F-statistic: 591.5 on 1 and 198 DF,  p-value: < 2.2e-16

The important thing to recognize the that the wrong model is almost as good as the true model. Moreover, with a data generating process like the one that underlies this simulation, it is entirely possible for the wrong model to have outperformed the true model. The more strongly correlated the variables are, the more likely that is to occur. I was actually in a situation once, where I wanted to use a covariate to absorb some of the error variance and boost power, but where I didn't care about that covariate--it wasn't germane substantively. I had several options available and they were all correlated with each other $r>.98$. I basically picked one at random and moved on, and it worked fine. I suspect I would have lost power burning two extra degrees of freedom if I had included the others as well. Again, my point is to think about what your goals are, and how you believe the variables are related to each other / why they are correlated, and use that to drive your strategy.

I agree that ridge regression is arguably better, but I think is also has two potential downsides. It is more complicated (requiring more statistical sophistication), and the resulting model is more difficult to interpret in my opinion.

I gather that perhaps the ultimate approach would be to fit a structural equation model. That would allow you to formulate the exact set of relationships you believe to be operative, including latent variables. However, I don't know SEM well enough to say anything about it here, other than to mention the possibility. (I also suspect it would be overkill in the situation you describe with just two covariates.)

Another option is that you can also combine the variables. This is done by standardizing both (i.e., turning them into z-scores), averaging them, and then fitting your model with only the composite variable. This would be a good approach when you believe they are two different measures of the same underlying construct. In that case, you have two measurements that are contaminated with error. The most likely true value for the variable you really care about is in between them, thus averaging them gives a more accurate estimate. You standardize them first to put them on the same scale, so that nominal issues don't contaminate the result (e.g., you wouldn't want to average several temperature measurements if some are Fahrenheit and some are Celsius). Of course, if they are already on the same scale (e.g., several highly-correlated public opinion polls), you can skip that step. If you think one of your variables might be more accurate than the other, you could do a weighted average (perhaps using the reciprocals of the measurement errors).

If your variables are just different measures of the same construct, and are sufficiently highly correlated, you really could just throw one out without losing much information. As an example, I was actually in a situation once, where I wanted to use a covariate to absorb some of the error variance and boost power, but where I didn't care about that covariate--it wasn't germane substantively. I had several options available and they were all correlated with each other $r>.98$. I basically picked one at random and moved on, and it worked fine. I suspect I would have lost power burning two extra degrees of freedom if I had included the others as well by using some other strategy. Of course, I could have combined them, but why bother? However, this depends critically on the fact that your variables are correlated because they are two different versions of the same thing; if there's a different reason they are correlated, this could be totally inappropriate.

As that implies, I suggest you think about what lies behind your correlated variables. That is, you need a theory of why they're so highly correlated to do the best job of picking which strategy to use. In addition to different measures of the same latent variable, some other possibilities are a causal chain (i.e., $X_1\rightarrow X_2\rightarrow Y$) and more complicated situations in which your variables are the result of multiple causal forces, some of which are the same for both. Perhaps the most extreme case is that of a suppressor variable, which @whuber describes in his comment below. @Macro's suggestion, for instance, assumes that you are primarily interested in $X$ and wonder about the additional contribution of $Z$ after having accounted for $X$'s contribution. Thus, thinking about why your variables are correlated and what you want to know will help you decide which (i.e., $x_1$ or $x_2$) should be treated as $X$ and which $Z$. The key is to use theoretical insight to inform your choice.

I agree that ridge regression is arguably better, because it allows you to use the variables you had originally intended and is likely to yield betas that are very close to their true values (although they will be biased--see here or here for more information). Nonetheless, I think is also has two potential downsides: It is more complicated (requiring more statistical sophistication), and the resulting model is more difficult to interpret, in my opinion.

I gather that perhaps the ultimate approach would be to fit a structural equation model. That's because it would allow you to formulate the exact set of relationships you believe to be operative, including latent variables. However, I don't know SEM well enough to say anything about it here, other than to mention the possibility. (I also suspect it would be overkill in the situation you describe with just two covariates.)

First, if your variables are sufficiently highly correlated, you simply don't lose much information by throwing one out. After all, that's pretty much what highly correlated means.

As that implies, I suggest you think about what lies behind your correlated variables. That is, you need a theory of why they're so highly correlated to do the best job of picking which option to use. For example, @Macro's suggestion (which I've also used) assumes that you are primarily interested in $X$ and wonder about the additional contribution of $Z$ after having accounted for $X$'s contribution. Thus, thinking in this way will help you decide which (i.e., $x_1$ or $x_2$) should be treated as $X$ and which $Z$.

First, (edited) in the real-world cases that I'm familiar with (data that I've worked with or read about) if one of a pair of very highly correlated variables had been thrown out, not much information would have been lost. (Be aware that this does not necessarily mean it's a good thing to do, I only mention this because it's often worth bearing in mind as a kind of baseline when you are faced with this situation.) This is likely due to the pattern of causal relationships that created the correlations between the variables in question and how they were related to the response variable. However, @whuber points out that it is completely possible for that not to be the case. I have no idea how often that kind of situation occurs versus the kinds that I'm more familiar with, but it's also worth bearing in mind.

As that implies, I suggest you think about what lies behind your correlated variables. That is, you need a theory of why they're so highly correlated to do the best job of picking which option to use. For example, @Macro's suggestion (which I've also used) assumes that you are primarily interested in $X$ and wonder about the additional contribution of $Z$ after having accounted for $X$'s contribution. Thus, thinking in this way will help you decide which (i.e., $x_1$ or $x_2$) should be treated as $X$ and which $Z$. (Edited) In line with what I said above, it's worth bearing in mind that the empirical / statistical properties of your model (e.g., goodness of fit) are likely to be quite similar whichever variable you choose for $X$ under many plausible situations (albeit perhaps not all). Thus, you want to use theoretical insight to inform your choice. For the sake of clarity, consider the following simulation:

library(MASS)
set.seed(8)
X = mvrnorm(n=200, mu=c(0,0), Sigma=rbind(c(1, .9),
                                          c(.9, 1)))
cor(X[,1], X[,2])     # [1] 0.9163053
y = 2 + 3*X[,1] + rnorm(200)

trueModel  = lm(y~X[,1])
wrongModel = lm(y~X[,2])

summary(trueModel)
...
Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  1.88429    0.07299   25.82   <2e-16 ***
X[, 1]       3.08083    0.06846   45.00   <2e-16 ***

Residual standard error: 1.03 on 198 degrees of freedom
Multiple R-squared: 0.9109, Adjusted R-squared: 0.9105 
F-statistic:  2025 on 1 and 198 DF,  p-value: < 2.2e-16 

summary(wrongModel)
...
Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   1.7586     0.1223   14.38   <2e-16 ***
X[, 2]        2.7916     0.1148   24.32   <2e-16 ***

Residual standard error: 1.729 on 198 degrees of freedom
Multiple R-squared: 0.7492, Adjusted R-squared: 0.7479 
F-statistic: 591.5 on 1 and 198 DF,  p-value: < 2.2e-16

The important thing to recognize the that the wrong model is almost as good as the true model. Moreover, with a data generating process like the one that underlies this simulation, it is entirely possible for the wrong model to have outperformed the true model. The more strongly correlated the variables are, the more likely that is to occur. I was actually in a situation once, where I wanted to use a covariate to absorb some of the error variance and boost power, but where I didn't care about that covariate--it wasn't germane substantively. I had several options available and they were all correlated with each other $r>.98$. I basically picked one at random and moved on, and it worked fine. I suspect I would have lost power burning two extra degrees of freedom if I had included the others as well. Again, my point is to think about what your goals are, and how you believe the variables are related to each other / why they are correlated, and use that to drive your strategy.