Fisher exact test with confounding variable

Question

I'm using a Fisher exact test to understand whether a Webpage-A is more likely to induce conversion compared to another Webpage-B.

               Webpage-A  Webpage-B
no-conversion        252        243
conversion          1557       1253

However, I know that the users are in different age ranges:

Young Webpage-A = 1668
Adults Webpage-A = 112
Elderly Webpage-A = 29

Young Webpage-B = 1355
Adults Webpage-B = 121
Elderly Webpage-B = 20

therefore, I under-sample my data-set, randomly selecting only:

1355 Young Webpage-A users
112 Adults Webpage-B users
20 Elderly Webpage-A users

in order to have a matching in terms of age ranges.

This changes the values of my matrix and the Fisher exact test gives a different p-value depending on the random seed I use to under-sample my data-set.

Is there a way to understand whether the age range is a confounding variable to be taken into account? If so, is the approach of under-sampling by age range correct? If so, how can interpret the different p-values I get depending from the random seed? Please, notice that the p-values go from less than 1% to up to 40% at least.

Answer 1

I would strongly recommended here to use a logistic regression and simply control for age by adding it as a covariate:

$logit(Conversion_i) = \beta_0 + \beta_1 Webpage_i + \beta_2 Age_i + \epsilon_i$

There are certainly other ways to control for a confounding variable, like matching, which is closer to what you are trying to do (note that you would have to account for the matched observations, for example with a paired test), but this will discard many observations and is not needed in this simple case.

You are implicitly making the assumption that age is the only (observed) confounder. If you have more information about users that could confound your true relationship or explain some variation, you should also include them in your model. The high sensitivity of the result to the random seed, and thus to the choice of which individuals exactly are sampled from the age classes, indicates that there is a lot of variation within age groups.