1
$\begingroup$

I've been working with JMP, and I have found that often I get a significant p value but a relatively small r. Is this supposed to happen?

$\endgroup$
2
  • 1
    $\begingroup$ Don't see why it would be impossible. With a large-enough sample size, any effect will be significantly different from 0. $\endgroup$ – Patrick Coulombe Mar 14 '14 at 1:47
  • $\begingroup$ just to clarify the sample size in this case was 99 $\endgroup$ – Max Pine Mar 14 '14 at 2:21
2
$\begingroup$

Yup. Just like @PatrickCoulombe said, it's normal to encounter such a situation with weak effects in large samples. Here's a demonstration in R (I assume you could do something similar in JMP):

X=data.frame(IV=sort(rep(letters[1:3],1000)),DV=sort(rnorm(3000))*.1+rnorm(3000))

This creates a data frame of 3000 observations organized into 3 categories (IV). DV is a randomly generated, standard normally distributed, continuous variable, to which I've added another such variable that's been sorted from low to high and had its variability scaled down to 10% of the original. This creates a weak signal in the otherwise random noise variation, such that the observations organized into the second and third groups will tend to be a little larger than the previous. Fitting a simple general linear model with ordinary least squares produces results more extreme than yours:

Call: summary(lm(DV~IV,X)). Results: multiple $R^2=.007, F_{(2,2997)}=10.22,p=.00004$.

Even with a smaller sample, this still works ($N=99$ isn't even all that small). In the following version, I'll just scale down the sorted component of DV to 20% instead of 10%, and produce only 99 cases:

set.seed(1);X=data.frame(IV=sort(rep(letters[1:3],33)),DV=sort(rnorm(99))*.4+rnorm(99))

Call: summary(lm(DV~IV,X)). Results: multiple $R^2=.079, F_{(2,96)}=4.15,p=.019$.

Seem familiar? With a smaller sample, it won't happen every time, so I set the random number generator's seed to 1 for reproducibility. With set.seed(2) for instance, $R^2=.006,p=.73$, somewhat more as you would expect, I imagine. In the case of a simple ANOVA, the $F$-test compares means across groups, and tests any evidence of differences in your sample against the null hypothesis of no differences in the population. The meaning of this test is therefore very different from the meaning of $R^2$, which represents how much outcome variation is related to the predictor(s). As in my first example, even 0.7% of the variation (let alone 10%!) can be "significantly" better than nothing.

$\endgroup$
1
  • 1
    $\begingroup$ +1, I agree that n=99 isn't that small. $\endgroup$ – Patrick Coulombe Mar 14 '14 at 2:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.