I'm using a Fisher exact test to understand whether a Webpage-A is more likely to induce conversion compared to another Webpage-B.

               Webpage-A  Webpage-B
no-conversion        252        243
conversion          1557       1253

However, I know that the users are in different age ranges:

Young Webpage-A = 1668
Adults Webpage-A = 112
Elderly Webpage-A = 29

Young Webpage-B = 1355
Adults Webpage-B = 121
Elderly Webpage-B = 20

therefore, I under-sample my data-set, randomly selecting only:

1355 Young Webpage-A users
112 Adults Webpage-B users
20 Elderly Webpage-A users

in order to have a matching in terms of age ranges.

This changes the values of my matrix and the Fisher exact test gives a different p-value depending on the random seed I use to under-sample my data-set.

Is there a way to understand whether the age range is a confounding variable to be taken into account? If so, is the approach of under-sampling by age range correct? If so, how can interpret the different p-values I get depending from the random seed? Please, notice that the p-values go from less than 1% to up to 40% at least.


1 Answer 1


I would strongly recommended here to use a logistic regression and simply control for age by adding it as a covariate:

$logit(Conversion_i) = \beta_0 + \beta_1 Webpage_i + \beta_2 Age_i + \epsilon_i$

There are certainly other ways to control for a confounding variable, like matching, which is closer to what you are trying to do (note that you would have to account for the matched observations, for example with a paired test), but this will discard many observations and is not needed in this simple case.

You are implicitly making the assumption that age is the only (observed) confounder. If you have more information about users that could confound your true relationship or explain some variation, you should also include them in your model. The high sensitivity of the result to the random seed, and thus to the choice of which individuals exactly are sampled from the age classes, indicates that there is a lot of variation within age groups.

  • $\begingroup$ This is how I would do it, but I think the point of the question is to generalize Fisher to include a covariate in the sense that, when we omit the covariate, the test is exactly Fisher, kind of like how ANCOVA with no covariate is ANOVA. $\endgroup$
    – Dave
    May 28, 2020 at 12:41
  • $\begingroup$ @stefgehrig should the Webpage be the binary treatment indicator? $\endgroup$
    – Newbie
    May 28, 2020 at 14:03
  • $\begingroup$ Also, I suppose that I shouldn't use the conversion for matching because that's the variable for which I want to apply the Fisher exact test. Please, could you clarify? $\endgroup$
    – Newbie
    May 28, 2020 at 14:21
  • $\begingroup$ Yes, the webpage is a binary treatment indicator. I don't understand the rest of the question. You do not need to modify the dataset in any way to run the logistic regression I outlined above. $\endgroup$
    – stefgehrig
    May 28, 2020 at 15:35
  • $\begingroup$ @stefgehrig, sorry, I thought that you were suggesting the logit for a propensity score matching. That's why I got confused. $\endgroup$
    – Newbie
    Jun 2, 2020 at 0:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.