2
$\begingroup$

I wonder how to generate such data, so that in single variable regression feature coefficient would be positive, and in multiple regression would be negative.

So I read several related questions on Cross Validated. As I understood, there are 2 possible cases for this desired situation: when features are correlated (collinearity) or when feature values are not orthogonal.


So I made a code that generates data for collinearity case:

set.seed(1)
xPositive = rnorm(100)
xNegative = xPositive / 3 + rnorm(100, 0, .15)
y = 5 * xPositive - 3 * xNegative + rnorm(100)
fitXNegative = glm(y~xNegative)
fixBothX = glm(y~xNegative+xPositive)
print(paste("Coefficient for xNegative when fitting only xNegative:", coef(fitXNegative)['xNegative']))
print(paste("Coefficient for xNegative when fitting both x'es:", coef(fixBothX)['xNegative']))
print(paste("That is because of correlation between x'es:", cor(xNegative, xPositive)))

The output is:

Coefficient for xNegative when fitting only xNegative: 9.57682149821626
Coefficient for xNegative when fitting both x'es: -2.85321980961991
That is because of correlation between x'es: 0.911618247307253

So the example works.


Similarly, I was trying to come up with non-orthogonality example. Here is my best try so far:

set.seed(1)
xPositive = runif(100, 100, 120)
xNegative = runif(100, 100, 120)
y = 13 * xPositive - 11 * xNegative
fitXNegative = glm(y~0+xNegative)
fitBothX = glm(y~0+xNegative+xPositive)
print(paste("Coefficient for xNegative when fitting only xNegative:", coef(fitXNegative)['xNegative']))
print(paste("Coefficient for xNegative when fitting both x'es:", coef(fixBothX)['xNegative']))

The output is:

Coefficient for xNegative when fitting only xNegative: 1.97024384743124
Coefficient for xNegative when fitting both x'es: -2.85321980961991

But as you may have noticed, I manually excluded the intercept from the model, because otherwise the coefficient doesn't change its sign. Also, if you center the data in this example:

xPositive = xPositive - 110
xNegative = xNegative - 110

then you effectively get rid of non-orhogonality and the coefficient then doesn't change its sign.


So, regarding this second non-orthogonality example I have 2 questions:

  1. I would like to come up with a better mechanism of data generation so that I could observe the effect of changing coefficient sign without the necessity of setting the intercept to zero.

  2. I am checking for non-orthogonality between features using the following formula:

    nonOrthogonalityMeasure = mean(xPositive * xNegative)

If that value approximately equals zero then I conclude that features are orthogonal. Is this a correct method?

  1. Are there other situations when the coefficient may change its sign, except for the aforementioned situations?
$\endgroup$
1
$\begingroup$

Uncorrelated:

  • In population: $\text{Cov}=\mathbb{E}XY-\mathbb{E}X \mathbb{E}Y=0$
  • In sample: $\widehat{\text{Cov}}=\frac{1}{n-1}\sum_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})=0$

Orthogonal:

  • In population: $\text{M}_{XY}=\mathbb{E}XY=0$
  • In sample: $\hat{\text{M}}_{XY}=\frac{1}{n}\sum_{i=1}^n x_i y_i=0$

The two coincide in population when $\mathbb{E}X=0$ and/or $\mathbb{E}Y=0$; they coincide in sample when $\bar{x}=0$ and/or $\bar{y}=0$.

The relevant feature for your question is correlation rather than orthogonality. When regressors are correlated (have non-zero correlation), their estimated coefficients in a multiple regression do not match their estimated coefficients from bivariate regressions; when they are uncorrelated, the coefficients match.

You may notice that centering the data by subtracting their sample means effectively turns an orthogonal pair of vectors to an uncorrelated pair of vectors; the non-central cross moment $\hat{\text{M}}_{XY}=0$ turns into the central cross moment $\widehat{\text{Cov}}=0$.

Regarding Question 1, here is a tip for generating orthogonal variables: running a regression of $y$ on $x$ will generate a residual $\varepsilon$ that is uncorrelated and also orthogonal (since it has a mean of zero) to $x$.

Regarding Question 2, the in-sample orthogonality measure defined above suggests that your approach is sensible (if I interpret correctly that xPositive and xNegative are two vectors and you want to check whether they are orthogonal).

$\endgroup$
  • $\begingroup$ You say "since what you do is turn your data from being orthogonal to being uncorrelated", but that's wrong. In my second example correlation is the same before and after centering and is equal to 0.017. Also if I understood you correctly you said that in my second example data is orthogonal BEFORE data centering. But mean(xPositive * xNegative) = 12178.32 $\endgroup$ – Yurii Feb 10 '16 at 8:46
  • $\begingroup$ I must have misunderstood what you did there. You started out saying "I was trying to come up with orthogonality example", so I thought you made up a pair of orthogonal vectors. Perhaps you could clarify (edit) that part of your question? I will edit the answer accordingly. $\endgroup$ – Richard Hardy Feb 10 '16 at 8:47
  • $\begingroup$ Oh, sure, my fault. I edited the questions, so 'orthogonality' is replaced with 'non-orthogonality'. I hope now it is less confusing. $\endgroup$ – Yurii Feb 10 '16 at 8:54
  • 1
    $\begingroup$ Are you sure you get rid of non-orthogonality by just subtracting the mean from each of the vectors? This looks like a special case but should not hold in general. Also, if you want the coefficient to change sign from multiple to bivariate regression, mere non-zero correlation will generally not suffice. I guess it's possible to derive an analytical condition for that, but I will not attempt that. $\endgroup$ – Richard Hardy Feb 10 '16 at 9:15
  • $\begingroup$ Well yes, after you said that I understood that I can imagine feature vectors where mere centering will not help to get rid of non-orthogonality. E. g. [1 2 150] and [3 4 71]. $\endgroup$ – Yurii Feb 10 '16 at 9:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.