# Hypothesis test for intervention with class imbalance before intervention. Alternative to McNemar

I want to assess an intervention to cure a disease. An example is described by the following matrix:

responses <- c("Present", "Absent")
matrix(c(101,59,121,33),nrow=2,dimnames = list("Before" = responses,"After" = responses))

After
Before    Present Absent
Present     101    121
Absent       59     33


This example was taken from the McNemar's Wikipedia page. McNemar may be useful to test equality of marginal probabilities for two different exams/tests, however for this particular before-and-after case ignoring a and d seems an incomplete analysis and seems potentially misleading or I am missing something. It is easier to explain with the following example:

M <- matrix(c(1,9,9,81),nrow=2,dimnames = list("Before" = responses,"After" = responses))
M %>% list(.,mcnemar.test(.),chisq.test(.))

[]
After
Before    Present Absent
Present       1      9
Absent        9     81

[]

McNemar's Chi-squared test

data:  .
McNemar's chi-squared = 0, df = 1, p-value = 1

[]

Pearson's Chi-squared test

data:  .
X-squared = 0, df = 1, p-value = 1


Neither the McNemar nor the Chi-square tests reject the null. For McNemar clearly the before and after marginal proportions are equal given that b and c are equal. For the Chi-Square the expected values are equal to the observed, hence the statistic is zero.

However there is a story to be told if we consider a and d and the class proportions before intervention. We can see that from a total of 10 patients with disease Present Before the intervention, 9 changed to Absent, i.e., 90% improved; while from a total of 90 patients initially Absent of disease, only 10% changed to present. This suggests an asymmetry in change.

Which test would support a claim about statistically significant difference in change considering proportions?

How about applying McNemar to the matrix normalised considering initial proportions (i.e., rows sum 100)?

[DISCLAIMER: Assessing whether the intervention was effective or not is hard in this case without a control group, however, being able to justify with stats what the data is telling is meaningful]

EXTRA: The Wikipedia page says that "the null hypothesis of "marginal homogeneity" would mean there was no effect of the treatment". This would be concluded for my later example. However if we replace the second row (Before-Absent) with 1,9, instead of 9,81, McNemar rejects and the conclusion would be the opposite, despite the ratio being the same, the only thing that changed is the sample size of the (Before-Absent) group.

Is the Wikipedia description misleading or am I missing something?

If I'm reading your example correctly, you have a trial with 100 patients. Before, 10 (10%) have the condition present, and after, 10 (10%) have the condition present. So, it doesn't sound like such a great treatment.

The problem with looking at the percentages the way you want to is this: Since the natural occurrence of the condition is small (10% here), a treatment causing that condition in 10% of healthy people causes just as much harm as removing it from 90% of people who already have it.

You might think of an analogy where adding a certain chemotherapy to the water supply would cure all 3 million Americans with prostate cancer, but cause another 3 million cases. You'd be saving 100% of prostate cancer sufferers, and only harming 1% of those without it!

These considerations become real issues in real cases of disease detection, prevention, and cure, because most diseases occur in low rates in the general population. So even if a test for a cancer has a low false positive rate, if the natural occurrence of that cancer is very low, the number of false positives can become high relative to the number of true positives. If a positive test result initiates potentially harmful actions (invasive procedures, anxiety, and so on), the benefits of the test need to be weighed against the likely negative consequences.

Bayes' theorem helps in thinking through these.

• One can argue that the treatment was positive cause it may have avoided 20 healthy to get the disease and cured 9. Of course there’s no evidence to claim this neither your point that the treatment caused 9 diseases, a control sample is needed, without it the analysis is lame and it is hard to conclude. So setting that aside, I want to find a test that considers the proportions, or an alternative as the one proposed by @a_statistician. Thanks for your post, I will add a disclaimer. – JPCampos Oct 24 '18 at 22:24
• My advice at this point is to think about what you are trying to show from the beginning. You want to compare 1) the rate at which the intervention causes disease in healthy individuals, vs. 2) the rate at which the intervention cures the disease in diseased indivs. Assuming the disease isn't affected by other factors during the trial, you could calculate these rates. But, is it meaningful to compare these rates for statistical difference? Practically speaking, how would you apply these rates, for example to the population as a whole or to a set of indivs testing positive for the disease? – Sal Mangiafico Oct 25 '18 at 9:13

"However if we consider a and d and the initial proportions we can see that from a total of 10 patients with disease Present Before the intervention, 9 changed to Absent, i.e., 90% improved; while from a total of 90 patients initially Absent of disease, only 10% changed to present. This suggests that the intervention had a positive impact."

What you said is probability of change. If you really want to compare the probability of change (present to absent (90%) vs absent to present (10%)), you can change the data to this. And then perform Pearson chi square test or Fisher's exact test.

          No change change
Present       1      9
Absent        81     9


Statistically, you can do this. But I am not sure if comparing the probability of change is reasonable, defensible, and it depends on your judgement.

• I agree with the last caveat: I wouldn't do something like this for this kind of problem. – Sal Mangiafico Oct 24 '18 at 13:39
• Statistically, I suspect applying a chi-square test of association to this table isn't allowed, because in the categories along the top, "change" means change from "Present" to "Absent" or vice-versa. The categories along the top are not conceptually distinct from the categories along the left. I haven't found any description of assumptions of this test that make this explicit, but I suspect I'm on to something. – Sal Mangiafico Oct 24 '18 at 14:31
• If you really want to test probability of change, in your case 90% vs 10%, the method I presented is correct. – user158565 Oct 24 '18 at 19:02
• This sounds like a good option :), I am also trying to wrap my head and convince myself there's no flaw in applying a chi-square to the matrix like that – JPCampos Oct 24 '18 at 23:38
• I would ask you to reconsider using chi-square like this. Change / Not-change are not states or categories the way that Alive / Dead are. Instead, Change / Not-change are relative to the categories on the left. So, here, in the first row Change means "Becomes healthy", and in the second row Change means "Becomes diseased". I don't think you should use a contingency table this way where the columns mean opposite things in the different rows. – Sal Mangiafico Oct 25 '18 at 9:01

I'm working this out as I go, so apologies for any inconsistencies. It would certainly need reliable back up (perhaps a reader will know of relevant resources or links) or a more complete working out before I'd act upon it.

Which test would support this conclusion that the intervention was effective for the later example?

How about applying McNemar to the matrix normalised considering initial proportions (i.e., rows sum 100)?

## What you are describing is basically the ratio of 1-Sensitivity to 1-Specificity.

1-Sensitivity is $$b/(a+b)$$

In your case this would be 1-Sensitivity of 0.9 - so that means 90% of those with the disease present will regress (recover) during treatment.

1-Specificity is $$c/(c+d)$$

In your case this would be 1-specificity of 0.1 - so that means 10% of those without the disease present will develop it during treatment.

The ratio will give you an estimate of the relative effect size for the treatment causing disease regression vs causing disease progression.

## Null and Alternative Hypotheses

In this scenario your null hypothesis is that the changes observed could be due to random chance. The alternative hypothesis is that the changes are greater than could be expected by random chance.

in this case the null (effect size = 1) is that $$1-sensitivity = (1-specificity)*(N_{present}/N_{total})*1$$ the alternative for a one sided test is $$1-sensitivity >> (1-specificity)*(N_{present}/N_{total})*1$$

For a two sided test one would consider whether the effect size is significantly lower than 1 also.

If the treatment is having an effect and if, in the absence of treatment, disease progression and regression is random. This means there is a positive pressure for patients with present, none for patients with absent. In this scenario there is a balance between the rate of regression vs the group imbalance. If these balance out the net effect will be 0. As many people will progress by random chance as regress by random chance regression plus active influence of treatment multiplied by its proportion.

Without accounting for proportions in the McNemars test

The null for would looks like $$1-sensitivity = (1-specificity) = 0.1$$ the alternative is that $$1-sensitivity = (1-specificity)*EffectSize >> 0.1$$

and this would fail because it effect size has to be >10 to even marginally exceed the null, never mind get to significance.

If we account for proportions:

The null for would looks like $$1-sensitivity = (1-specificity)*(N_{present}/N_{total}) = 0.01$$ the alternative is that $$1-sensitivity = (1-specificity)*(N_{present}/N_{total})*EffectSize >> 0.01$$

So it would have a higher probability of succeeding

## However

A big issue is that with vastly different sample numbers for each condition, the statistical power is much lower for the condition with the lower proportion, of the risk of accepting a false alternative hypothesis is higher. A study would need to be appropriately powered in order to apply a correction for imbalance in the statistical analysis.

• what is the distribution under the null @ReneBt? – JPCampos Oct 24 '18 at 23:39
• That's one of the many questions that would need addressed before anyone would use it, part of that complete working out mentioned. I initially dismissed the idea, but working through it has made me think its worth investigating more fully but would need a full proof and assessment of edge cases. If you needed to stand behind using it, there would need to be a peer reviewed scientific article. If it's for your own risk assessment, for making a decision then it's up to you the level of proof needed. All statistics boils down to is risk assessment – ReneBt Oct 25 '18 at 4:31