13
$\begingroup$

The other day I had a consultation with an epidemiologist. She is an MD with a public health degree in epidemiology and has a lot of statistical savvy. She mentors her research fellows and residents and helps them with statistical issues. She understands hypothesis testing pretty well. She had a typical problem of comparing two groups to see if there is a difference in their risk related to getting congestive heart failure (CHF). She tested the mean difference in the proportion of subjects getting CHF. The p-value was 0.08. Then she also decided to look at the relative risk and got a p-value of 0.027. So she asked why is one significant and the other not.

Looking at 95% two-sided confidence intervals for the difference and for the ratio she saw that the mean difference interval contained 0 but the upper confidence limit for the ratio was less than 1. So why do we get inconsistent results?

My answer while technically correct was not very satisfactory. I said "These are different statistics and can give different results. The p-values are both in the area of marginally significant. This can easily happen."

I think there must be better ways to answer this in laymen's terms to physicians to help them understand the difference between testing relative risk vs absolute risk. In epi studies, this problem comes up a lot because they often look at rare events where the incidence rates for both groups are very small and the sample sizes are not very large.

I have been thinking about this a little and have some ideas that I will share. But first I would like to hear how some of you would handle this. I know that many of you work or consult in the medical field and have probably faced this issue. What would you do?

Improve this question
$\endgroup$
4
  • $\begingroup$ Do the models include other covariates besides the group effect? $\endgroup$
    – onestop
    Commented Jun 1, 2012 at 11:23
  • $\begingroup$ @onestop There are covariates that they are interested in looking at but the actual test was only comparing the main effect. If you would like to comment supposing that the test was based of a regression model or event assume we had time to event data to fit a Cox regression model feel free to comment. I would love to hear your insights. My question is addressed to the general problem and not just the specific example. $\endgroup$
    – Michael R. Chernick
    Commented Jun 1, 2012 at 11:53
  • $\begingroup$ I meant, was the test comparing the main (group) effect adjusted for covariates, or unadjusted? If unadjusted, then it might be helpful to give us the 2×2 table, or a similar one, to focus ideas. $\endgroup$
    – onestop
    Commented Jun 1, 2012 at 14:22
  • $\begingroup$ Unadjusted for these particular tests. $\endgroup$
    – Michael R. Chernick
    Commented Jun 1, 2012 at 14:28

2 Answers 2

Reset to default
8
$\begingroup$

Well, from what you've already said, I think you've got most of it covered but just need to put it in her language: One is a difference of risks, one is a ratio. So one hypothesis test asks if $p_2 - p_1 = 0$ while the other asks if $\frac{p_2}{p_1} = 1$. Sometimes these are "close" sometimes not. (Close in quotes because clearly they aren't close in the usual arithmetic sense). If the risk is rare, these are typically "far apart". e.g. $.002/.001 = 2$ (far from 1) while $.002-.001 = .001$ (close to 0); but if the risk is high, then these are "close": $.2/.1 = 2$ (far from 0) and $.2 - .1 = .1$ (also far from 0, at least compared to the rare case.

Improve this answer
$\endgroup$
7
  • 2
    $\begingroup$ You have one of my ideas in there, when number are small which is common in studying low incidence rates differences look small but ratios still look large. Your numerical example is very compelling. I am tempted to add something about the stability of the estimates under the null hypothesis. For some this may be too technical but at her level of sophistication maybe not. Suppose the two populations have nomral distributions mean zero and known common variance. Then the normalized difference is N(0,1) under the null hyothesis giving a very stable test statistic. $\endgroup$
    – Michael R. Chernick
    Commented Jun 1, 2012 at 11:43
  • 1
    $\begingroup$ But under these assumptions the ratio has a Cauchy distribution and can be very large. Maybe this argument needs modification since the incidence rates have to be positive and possibly the distribution is very skewed. I guess what I want is an example showing the difference has a very stable distribution and the ratio does not especially because the sample size is small and the denominator can get very close to 0. Anyone got a good illustrative example? $\endgroup$
    – Michael R. Chernick
    Commented Jun 1, 2012 at 11:49
  • $\begingroup$ @Peter Did you mean to write three $p_i$s not two? If so could you define your notation? $\endgroup$
    – onestop
    Commented Jun 1, 2012 at 14:17
  • $\begingroup$ I think he meant p1 when he wrote p0. Just a basic error. Having three ps in this context makes no sense. $\endgroup$
    – Michael R. Chernick
    Commented Jun 1, 2012 at 15:53
  • 1
    $\begingroup$ I made the change for Peter. Scream at me if I did something wrong! $\endgroup$
    – Michael R. Chernick
    Commented Jun 1, 2012 at 15:55
9
$\begingroup$

Mind that in both tests, you test a completely different hypothesis with different assumptions. The results are not comparable, and that is a far too common mistake.

In absolute risk you test whether the (average) difference in proportion differs significantly from zero. The underlying hypothesis in the standard test for this assumes that the differences in proportion are normally distributed. This might hold for small proportions, but not for large. Technically you calculate the following conditional probability :

$$P( p_1 - p_2 = 0 | X )$$

with $p_1$ and $p_2$ the two proportions, and $X$ your explanatory variable. This is equivalent to testing the slope $b$ of the following model :

$$p = a + b\cdot X + \epsilon$$

where you assume that $\epsilon \sim N(0,\sigma)$.

In relative risk you do something completely different. You test the odds of having a positive outcome based on the explanatory variable $X$. So you calculate

$$P\left( \log\left(\frac{p_1}{p_2}\right) = 0 | X \right)$$

which is equivalent to testing the slope in the following logistic model:

$$\log\left(\frac{p}{1-p}\right) = a + b\cdot X + \epsilon$$

with $\log(\frac{p}{1-p})$ being the log of the odds. Note that this hypothesis is formulated in terms of the odds, and not proportions! So the assumptions of the model are also formulated in terms of the odds (or more exactly, the log of the odds). You're testing a different hypothesis.

The reason why this makes a difference is given in Peter Flom's answer: a small difference in absolute risks can lead to a big value for the odds. So in your case it means that the proportion of people getting the disease don't differ substantially, but the odds of being in one group is significantly larger than the odds of being in the other group. That is perfectly sensible.

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ I think we all so far agree that the main reason for the problem is that small differences in absolute risk can leads to large differences in relative risk. After all .2 to.1 has the same relative risk as 0.0002 to 0.0001. I think this is the message we can bring home to the layperson. Your explanation is great for statisticians but I am not sure that it would be easily understood by a layperson and one could say "So what if you are testing a different hypothesis. $\endgroup$
    – Michael R. Chernick
    Commented Jun 1, 2012 at 13:44
  • $\begingroup$ You are still trying to determine where or not the rates are different. So even though the hypotheses are different the results should be consistent. After all p1-p2=0 is the same as p1/p2 =1." So I think the fact that the hypotheses are different misses the point and is not a satisfactory explanation. $\endgroup$
    – Michael R. Chernick
    Commented Jun 1, 2012 at 13:44
  • $\begingroup$ @MichaelChernick I was about to say that proportion differences are conditional, and odds ratio is not. But that's not the case, both give exactly the same result after transposing the table (in the case of a 2X2 table). I've been running some simulations, but I can't force the p-values of prop.test (or chisq.test as it is equivalent in the 2x2 case) and fisher.test to be more than 0.005 apart. So I wonder which tests she used... $\endgroup$
    – Joris Meys
    Commented Jun 1, 2012 at 15:15
  • $\begingroup$ It would either be chi square or Fisher's test. Most likely Fisher's test because she knows in small samples that the chi square approximatation is not good. When I do statistics for them I use SAS. She did her work using STATA. I can probably dig up the actual table. $\endgroup$
    – Michael R. Chernick
    Commented Jun 1, 2012 at 15:58
  • 2
    $\begingroup$ One additional consideration, since we're getting into this: $log(\frac{p_1}{p_0}) = log(p_1)-log(p_0)$ which is clearly different from $p_1 - p_0$ and it is more different precisely when the p are small - that is, risk is slight. But I was trying to keep my first answer ASAP (that's As simple as possible!) $\endgroup$
    – Peter Flom
    Commented Jun 1, 2012 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.