# Why is relative risk not valid in case control studies?

I'm getting my information from Overview of Analytic Studies. It says that in a case-control study, you can not compute the probability of disease in each exposure group because you don't have the total amount of people in your population. But you just made a new population for the case control study - why not use this total as the population total? Of course it's a sample of the population you're really interested in so that introduces sampling bias and error but that is true for the odds ratio as well. So you'd have to say it's the approximation of the true relative risk.

Another argument I've read is that in a case-control study, you haven't really taken a cross-section of the population because you started with a fixed amount of people with the same outcome (=diseased), but the amount of individuals in this group (disregarding noise) is irrelevant: if this group is bigger, the amount of of exposed people grows with the same multiplier as the amount of unexposed people. So when calculating the relative risk (risk exposed group / risk in unexposed group) both the numerator and the denominator grow with the same amount, which then makes no difference in the value of the fraction.

Can anyone explain what I'm missing?

• It would represent the relative risk of being included in the sample as a case. The relative risk of the actual outcome could be different. Some say it's not the worst form of bias out there, and the inference is essentially valid anyway. Ollie Miettenen (1974?) has a good paper on this, but I lack the citation atm. Dec 28, 2022 at 3:33

I'll try to explain this more intuitively and with an illustration.

The risk ratio and the odds ratio can be interpreted and calculated as probabilities. These probabilities depend on the study design. Before I start writing formulas, let me be clear with some symbols.

$$X$$ = outcome

$$Y$$ = exposure

$$\neg{}X$$ = no outcome

$$P(X|Y)$$ = Bayesian probability of X happening, given that Y happened

# Risk

For example if you know the complete information from a population and you want to compute the risk (probability) of an outcome, given an exposure, you would write: $$Risk_{pop} = P(X|Y)$$ And the risk ratio between having an outcome, given an exposure, and having an outcome with no exposure, would be: $$RR_{pop} = \frac{P(X|Y)}{P(X|\neg{Y})}$$

Now, if you are sampling from a population, things get a little different, depending on the sampling design. That's because when you sample, you're drawing from a population with a specific probability. If you sample people based on their exposure status (cohort design), and then wait until you see the outcome, you would have $$Risk_{cohort}=\frac{P(X|Y)}{P(X|Y)+P(\neg{X}|Y)}=\frac{P(X|Y)}{1}=Risk_{pop}$$

Which is precisely the same as calculating the population risk. Then if you try calculating the risk ratio, since $$Risk_{cohort} = Risk_{pop}$$, then $$RR_{cohort} = RR_{pop}$$. So a cohort study has the perfect design for calculating the population risk.

However, if you sampled people based on their outcome status (case-control design), and then checked whether they were exposed or not, you would get a very different probability, that is the probability of finding an exposure, given that you know the outcome: $$Risk_{case-control}=\frac{P(Y|X)}{P(Y|X)+P(Y|\neg{X})}\ne{}P(X|Y), Risk_{case-control}\ne{}Risk_{pop}$$

and

$$RR_{case-control} = \frac{\frac{P(Y|X)}{P(Y|X)+P(Y|\neg{X})}}{\frac{P(\neg{Y}|X)}{P(\neg{Y}|X)+P(\neg{Y}|\neg{X})}}\ne{}\frac{P(X|Y)}{P(X|\neg{Y})}, RR_{case-control}\ne{}RR_{pop}$$

Therefore, you are not calculating the risk in a case-control study, because the probabilities are not the same.

# Odds

The odds of something happening is the probability of it happening divided by the probability of it not happening. For example, you would have 4 times more chance (odds) of winning than of losing if the probability of winning was 80%, because you would divide 80% by 20%. So the chance of an outcome, if you were exposed, would be:

$$Odds_{pop} = \frac{P(X|Y)}{P(\neg{X|Y})}$$ And the Odds Ratio would be the ratio between the odds of cancer if you smoked, and the odds of cancer, if you didn't smoke.

$$OR_{pop} = \frac{\frac{P(X|Y)}{P(\neg{X|Y})}}{\frac{P(X|\neg{}Y)}{P(\neg{X|\neg{}Y})}}$$

## Sample odds

If you were doing a case-control study, in which the Odds Ratio would be the choice for measuring the effect size, you would be calculating this:

$$OR_{case-control} = \frac{\frac{\frac{P(Y)P(X|Y)}{P(X)}}{\frac{P(Y)P(\neg{}X|Y)}{P(\neg{}X)}}}{\frac{\frac{P(\neg{}Y)P(X|\neg{}Y)}{P(X)}}{\frac{P(\neg{}Y)P(\neg{}X|\neg{}Y)}{P(\neg{}X)}}} = \frac{\frac{\frac{1.P(X|Y)}{1}}{\frac{1.P(\neg{}X|Y)}{1}}}{\frac{\frac{1.P(X|\neg{}Y)}{1}}{\frac{1.P(\neg{}X|\neg{}Y)}{1}}} = \frac{\frac{P(X|Y)}{P(\neg{}X|Y)}}{\frac{P(X|\neg{}Y)}{P(\neg{}X|\neg{}Y)}} = OR_{pop}$$

I won't write here the equation for the Odds Ratio in a cohort study, because it would be exactly the same as the population odds ratio, therefore they are also the same. Therefore, the odds ratio is an effect size measure that is adequate for both case-control and cohort designs, because they all measure the same thing.

# Simulation example

Now what would happen if you indeed tried to calculate a RR from a case-control design, what would happen?

This figure is the result of a simulation of a population of 2 million people, in which 20% smoked, 2% of the smoking population had cancer and 1% of the non smoking population had cancer. I simulated a cohort and a case-control design, with adequate sample sizes, and repeated the estimates 40 times in each case, for each effect size calculation. The code can be found here.

You can see that the distribution of the effect sizes are all similar for both study designs, when you are computing the adequate measures. However, when computing the RR in a case-control study, the distribution is very different from the others, never getting close to the true risk.

To obtain the relative risk you have to know the risk for each level of the exposure. If you sample people with each level of the exposure then you can estimate their risk of disease. You can then compare those risks which is what you are interested in. In a case control study you do not do that, you ascertain a number of cases in some way (usually not on the basis of a formal sampling strategy) and then find some controls who are similar to the cases (for some meaning of similar) except in their disease status. They are unlikely to be representative of the general population.

• Yeah so a case control study doesn't have perfect sampling, now why does that invalidate calculating the relative risk? What is it about odds ratio that makes it perfect for imperfect sampling? Apr 30, 2017 at 14:48
• You cannot calculate the relative risk because you don't know the probability $P(Y|X=1)$ and $P(Y|X=0)$ (that you could only derive if you knew the real distribution either of $Y$ or of $X$). On the contrary, since the odds ratio does depend on the distribution of $Y|X$ $(P(Y=1|X=1)$ and $P(Y=1|X=0))$, but equivalently of $X|Y$ $(P(X=1|Y=1)$ and $P(X=1|Y=0))$, we can calculate it, since the distribution of $X|Y$ is precisely what we observe. Mar 20, 2018 at 22:18

Succintly: Risk of X is a measure of probability of X over a period of time, however case-control designs measure X (case status in each subject) at a single point in time, and there is therefore no probability of X over a period time to be measured.

Inspired by Federico's comment, here is my take on formalizing the statistics:

Using the contingency table notation in Wikipedia:

                | Intervention (I)      | Control (C)
----------------|-----------------------|-------------
Events (E)      | IE                    | CE
Non-events (N)  | IN                    | CN


$$\textrm{Odds Ratio}= \frac{P(I\cap E)/P(I\cap N)} {P(C\cap E)/P(C\cap N)}= \frac{P(I| E)/P(I| N)} {P(C| E)/P(C| N)}$$

whereas

$$\textrm{Relative Risk}= \frac{P(I\cap E)/P(I)} {P(C\cap E)/P(C)}= \frac{P(I| E)/P(I)} {P(C| E)/P(C)}$$

The key difference being, for odds ratio, one needs to know the conditional probabilities $$P(I| N), P(C| N)$$, which are known in case control studies; whereas for relative risk, $$P(I), P(C)$$ are not known from a retrospective study design.

• Cool! Is there also a way to describe it more intuitively? Mar 29, 2020 at 23:12

It may help to think of both cohort and case-control studies being limited, just in opposite ways. A case-control study cannot give you risk of disease, but a typical cohort study where the numbers exposed and not exposed are fixed (e.g. 1000 treated vs 1000 not treated) cannot give you the probability of exposure (it's not really a "risk" of exposure, because studies usually require the exposure to occur before the outcome). A cross-sectional study can give you both the probability of exposure and the probability of outcome (though, as Alexis mentioned below in a comment, these are prevalences not risks in a cross sectional study. The math is the same.).

This is because risk is calculated from probabilities, and those probabilities in your study need to match what's in the actual population in order for your study to give good estimates. The moment you fix these probabilities to something other than reality, that limits what can be calculated from your study. For example, we force the probability of disease to 50% when we do a 1:1 case-control study. We force the probability of exposure to 50% when we do a 1:1 cohort study. These rarely match the true proportion in the population.

Here's an example. The true probabilities per group are shown in the table below from a cross-sectional study.

$${Disease}^+$$ $${Disease}^-$$ $$Total$$ $$P({Disease}^+)$$
$${Exposure}^+$$ 500 1500 2000 25%
$${Exposure}^-$$ 750 1500 2250 33%
$$Total$$ 1250 3000 4250 29%
$$P({Exposure}^+)$$ 40% 50% 47%

In a cohort study, you are fixing the number in each exposure group, say 1200 in each group. Thus, your study can't estimate the probability of being exposed anymore: you forced it to be 50%. Notice how the risk of disease is unchanged (far column), but the probabilities of exposure are all different now.

$${Disease}^+$$ $${Disease}^-$$ $$Total$$ $$P({Disease}^+)$$
$${Exposure}^+$$ 300 900 1200 25%
$${Exposure}^-$$ 400 800 1200 33%
$$Total$$ 700 1700 2400 29%
$$P({Exposure}^+)$$ 43% 53% 50%

In a case-control study, you are fixing the number of cases and controls, say 1200 in each group. Thus, your study can't estimate the probability of disease (risk). Notice how the probability of exposure is the correct, but the risk of disease is not.

$${Disease}^+$$ $${Disease}^-$$ $$Total$$ $$P({Disease}^+)$$
$${Exposure}^+$$ 480 600 1080 44%
$${Exposure}^-$$ 720 600 1320 55%
$$Total$$ 700 1200 2400 50%
$$P({Exposure}^+)$$ 40% 50% 47%

## Why?

RR is based on risks, and risks are population probabilities. We often estimate these probabilities using counts from our studies, but those estimates need to be valid approximations.

By definition, $$RR=\dfrac{Risk_{{Exposure}^+}}{Risk_{{Exposure}^-}},$$

where $$Risk_{{Exposure}^+}=P({Disease}^+|{Exposure}^+)$$, the probability of disease given that one is exposed,

and $$Risk_{{Exposure}^-}=P({Disease}^+|{Exposure}^-)$$, the probability of disease given one is not exposed.

Here's a generic cohort study. We picked $$b$$ and $$d$$ (e.g. 1200 people each), and then we measure $$a$$ and $$c$$ in the study (e.g. 300 and 400 respectively). I've left the rest of the table blank, because it's unneeded.

$${Disease}^+$$ $${Disease}^-$$ $$Total$$
$${Exposure}^+$$ $$a$$ $$b$$
$${Exposure}^-$$ $$c$$ $$d$$
$$Total$$

Looking at the table, you can estimate $$Risk_{{Exposure}^+}=P({Disease}^+|{Exposure}^+)\approx \frac{a}{b}$$, because $$b$$ is a constant, we measured $$a$$ in relation to $$b$$, and we do not need anything from the rest of the table. If I wind up picking a different $$b$$, then $$\frac{a}{b}$$ will stay roughly the same, because $$a$$ depends on $$b$$. You could view Row 1 as its own little experiment, independent of Row 2. Row 2 could have $$d$$ = 1 person or 1 billion people; it wouldn't change $$a$$, $$b$$, or the risk estimate. Similarly for $$Risk_{{Exposure}^-}\approx \frac{c}{d}$$.

[Delving into the statistical weeds, when $$b$$ and $$d$$ are fixed (aka given), we have two independent binomial distributions, $$Bin(n, p)$$. The expected value of a binomial distribution is $$E[Bin(n, p)]=np$$. For the exposed group, $$n=b$$ and $$p=Risk_{{Exposure}^+}$$. Since $$a$$ is our estimate of the expected value, we have $$a\approx E[Bin(b,Risk_{{Exposure}^+})]=b\cdot Risk_{{Exposure}^+}$$. Therefore, $$Risk_{{Exposure}^+}\approx \frac{a}{b}$$. Similarly for the non-exposed group.]

Now think about a case-control table. Here, we're given $$y$$ and $$z$$ as sample sizes for diseased and not diseased people, and from them we measure the number exposed, $$w$$ and $$x$$. I've added more detail to match the information we had for the cohort study.

$${Disease}^+$$ $${Disease}^-$$ $$Total$$
$${Exposure}^+$$ $$w$$ $$x$$ $$w+x$$
$${Exposure}^-$$ $$y-w$$ $$y+z-w-x$$
$$Total$$ $$y$$ $$z$$ $$y+z$$

The simplest "risk" to calculate here is the probability of exposure given disease status: $$P({Exposure}^+|{Disease}^+)\approx \frac{w}{y}$$. Compare to the cohort study, where the simplest risk was the risk of disease given exposure: $$P({Disease}^+|{Exposure}^+)\approx \frac{a}{b}$$. The previous answers have explained why $$\frac{w}{y}\neq \frac{a}{b}$$.

I suspect your question is actually: why can't we estimate $$Risk_{{Exposure}^+}$$ with $$\frac{w}{w+x}$$?

Unlike in the cohort design, we aren't given $$w+x$$; instead, we're given $$y$$ and $$z$$. Both $$w$$ and $$x$$ are measured quantities, and they are measured depending on $$y$$ and $$z$$. Now, we don't have a separate little experiment in the top row; instead, it's inextricably linked to the bottom Total row. We have to take that bottom row into account.

Remember that we picked $$y$$ and $$z$$, so we could pick any other numbers. Say we pick $$z^*$$, instead, where $$z^*\neq z$$, and from that we measure $$x^*$$ instead of $$x$$. Since $$z^*\neq z$$, then we would expect that $$x^*\neq x$$. Therefore, $$w+x^*\neq w+x$$ and so $$\frac{w}{w+x^*}\neq \frac{w}{w+x}$$. Thus we would get a different "risks" simply by choosing different sample sizes. The true risk, of course, does not change based on our selections.

$${Disease}^+$$ $${Disease}^-$$ $$Total$$
$${Exposure}^+$$ $$w$$ $$x^*$$ ($$\neq x$$) $$w+x^*$$
$${Exposure}^-$$ $$y-w$$ $$y+z^*-w-x^*$$
$$Total$$ $$y$$ $$z^*$$ ($$\neq z$$) $$y+z^*$$

[Back into the statistics, unlike $$b$$, $$w+x$$ is not constant. Therefore, the distribution in the top row is not $$Bin(w+x,Risk_{{Exposure}^+})$$. Since we cannot measure directly the risk, we'd have to use Bayes Theorem:$$Risk_{{Exposure}^+}=P({Disease}^+|{Exposure}^+)=\dfrac{P({Exposure}^+|{Disease}^+)P({Disease}^+)}{P({Exposure}^+)}.$$ We can estimate $$P({Exposure}^+|{Disease}^+)\approx \frac{w}{y}$$, because the binomial distribution holds. However, we cannot estimate $$P({Disease}^+)\not\approx \frac{y}{y+z}$$, because we chose $$y$$ and $$z$$, rather than measuring them from the experiment. Often we choose 1:1 groups, e.g. 1000 cases and 1000 controls. This would make $$\frac{y}{y+z}=0.5$$ even when the actual disease probability could be different. Similarly, we cannot estimate $$P({Exposure}^+)\not\approx \frac{w+x}{y+z}$$, because that requires having the correct mix of cases and control.

E.g. these two tables have the same $$P({Exposure}^+|{Disease}^+)=1$$ and $$P({Exposure}^+|{Disease}^-)=0$$, but different $$P({Exposure}^+)$$, 50% and 20% respectively.

$${Disease}^+$$ $${Disease}^-$$ $$Total$$
$${Exposure}^+$$ 100 0 100
$${Exposure}^-$$ 0 100 100
$$Total$$ 100 100 200
$${Disease}^+$$ $${Disease}^-$$ $$Total$$
$${Exposure}^+$$ 100 0 100
$${Exposure}^-$$ 0 400 400
$$Total$$ 100 400 500

If you know the disease and exposure probabilities from previous studies, then you can calculate risk and RR using them; but that information does not come from your case-control study (and isn't in the tables above).]

• Prevalence ≠ incidence rate. It is needlessly confusing to open with the intimation that they are synonyms. Dec 24, 2022 at 1:49
• Agreed. I removed the references to prevalence. They are not the same, despite often using the same models/math. Dec 28, 2022 at 2:58
• That's good, but you also have an erroneous assertion in the next sentence: "but a cohort study cannot give you the probability of exposure" Cohort designs can most certainly be designed to sample both exposure (or change in exposure) and change in outcome in a representative fashion. While it is valid to either sample exposed and unexposed groups separately in a cohort design, or to use an exposure-defined cohort (e.g., none are exposed at start of study time, but some are observed to become exposed), 'cohort design' in no way precludes representative samples of exposure as well as outcome. Dec 28, 2022 at 6:11
• Corrected again. Dec 30, 2022 at 21:17
• But if you know the proportion of the population having the disease and the proportion receiving the treatment... How do you calculate the RR from the OR in a case-control study? I've found the equation RR=OR/(1-p-p·OR) where p is the risk in the control group. But I don't know if it's always true.
– skan
Jun 3, 2023 at 21:58