Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm working with some real world data and the regression models are yielding some counterintuitive results. Normally I trust the statistics but in reality some of these things can not be true. The main problem that I am seeing is that an increase in one variable is causing an increase in the response when, in fact in reality, they must be negatively correlated.

Is there a way to force a specific sign for each of the regression coefficients? Any R code to do this would be appreciated as well.

Thanks for any and all help!

share|improve this question
8  
1  
Might be related here –  Matthew Jun 26 at 16:58

3 Answers 3

  1. beware the distinction between the marginal correlation and the partial correlation (correlation conditional on other variables). They may legitimately be of different sign.

    That is $\text{corr}(Y, X_i)$ may in fact be negative while the regression coefficient in a multiple regression is positive. There is not necessarily any contradiction in those two things. See also Simpson's paradox, which is somewhat related (especially the diagram). In general you cannot infer that a regression coefficient must be of one sign merely based on an argument about the marginal correlation.

  2. Yes, it's certainly possible to constrain regression coefficients to be $\geq 0$ or $\leq 0$*. There are several ways to do so; some of these can be done readily enough in R, such as via nnls. See also the answers to this question which mention a number of R packages and other possible approaches.

    However I caution you against hastily ignoring the points in 1. just because many of those are easily implemented.

    * (you can use programs that do non-negative to do non-positive by negating the corresponding variable)

share|improve this answer

There may well be such a way but I would say that it is not advisable in your circumstances.

If you have a result that is impossible either:

1) There is a problem with your data 2) There is a problem with your definition of "impossible" or 3) You are using the wrong method

First, check the data. Second, check the code. (Or ask others to check it). If both are fine then perhaps something unexpected is happening.

Fortunately for you, you have a simple "impossibility" - you say two variables cannot be positively correlated. So, make a scatter plot and add a smoother and see. A single outlier might cause this; or it might be a nonlinear relationship. Or something else.

But, if you are lucky, you've found something new. As my favorite professor used to say "If you're not surprised, you haven't learned anything".

share|improve this answer
    
(+1 to both Peter and Glen) @JRW - If you do fix the sign, I'd like to be a fly on the wall when you try to explain to your audience the coefficient you "obtained," and its confidence interval. Moreover, they might legitimately ask, Did you fix the sign and/or range of others? If not, why not? –  rolando2 Jun 27 at 18:42

To answer your specific question, you can try the nnls package which does least squares regression with non-negative constraints on the coefficients. You can use it to get the signs you want by changing the signs of the appropriate predictors.

By the way, here is a very simple way to create a dataset to demonstrate how it is possible to have positive correlations and negative regression coefficients.

> n <- rnorm(200)
> x <- rnorm(200)
> d <- data.frame(x1 = x+n, x2= 2*x+n, y=x)
> cor(d)
      x1        x2         y
 x1 1.0000000 0.9474537 0.7260542
 x2 0.9474537 1.0000000 0.9078732
 y  0.7260542 0.9078732 1.0000000
> plot(d)
> lm(y~x1+x2-1, d)

Call:
lm(formula = y ~ x1 + x2 - 1, data = d)

Coefficients:
x1  x2  
-1   1  
share|improve this answer
    
I just toyed arround with this nnls package a bit. Is there any way to get an adjusted R-squared value (or something equivalent), or would I have to try and calculate it myself somehow? –  JRW Jun 26 at 21:10
    
Why are you ignoring the advice you got from Glen and myself? –  Peter Flom Jun 27 at 21:28

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.