0
$\begingroup$

I saw this thread (Effect of binary variables on binary outcomes) and I saw the user used Fisher Exact Test. My research is as follows, does a high quality photo lead to the sale of a product in an online webshop (X = quality of photo; 0 = bad, 1 = good; Y = sale; 0 = no, 1 = yes).

I am not familiar with the Fisher Exact Test and I see on this site (https://www.r-bloggers.com/barnard%E2%80%99s-exact-test-%E2%80%93-a-powerful-alternative-for-fisher%E2%80%99s-exact-test-implemented-in-r/) that Barnard's test might be more suitable but I cannot understand the reasoning behind it. Can someone explain it to me in more layman terms?

Could you recommend any other tests/models (for instance a decision tree)? And why would it be better? Would really like to understand this. And can I introduce control variables like the price of a product?

$\endgroup$
  • 1
    $\begingroup$ Clarification: ¿what are the frequency sizes in the 2x2 table? My response will depend on the general size (<5 is very small, <15 is small, >50 is large). $\endgroup$ – Gregg H Mar 23 '18 at 3:58
  • $\begingroup$ A lot larger than 50 ;) $\endgroup$ – Stephen Theunissen Apr 9 '18 at 12:58
1
$\begingroup$

There are two asymptotic approaches (they work better as $N$ gets big... towards 20 and beyond, say), depending on whether your binary outcomes ($O$) are paired (i.e. the explanatory variable ($E$) indicates two different times of measurement in the same individual, or in a pair of matched individuals), or whether they are independent across the two values of $E$ (e.g., sampling from group $E=0$ and measuring your binary outcome in them, and independently sampling from group $E=1$ and measuring your outcome in them).

Paired test for association between binary variables
In the first case, using paired data, McNemar's test provides a way to test $N$ pairs of people for association between $E$ and $O$. Each pair contains one individual with $E=0$, and one individual with $E=1$. Because your data have this pairing, there are four possible kinds of pairs (the sum of all four cells equals $N$:

Observed counts of pairs
------------------------------
O=0 & E=0     |  O=0 & E=1 (r)
--------------+---------------
O=1 & E=0 (s) |  O=1 & E=1

McNemar's test works as follows:

  1. Formulate your hypotheses:
    $H_{0}: P(r) = P(s)$
    $H_{A}: P(r) \ne P(s)$

    where $r$ is the number of pairs where $O=1$ and $E=1$, and $P(r)$ is the probability of observing such a pair, and $s$ is the number of pairs where $O=1$ and $E=0$, and $P(s)$ is the probability of observing such a pair. Notice that $r$ and $s$ are discordant pairs.

  2. Calculate a test statistic from your data (the absolute value and the "$-1$" in the numerator are a continuity correction per Edwards):

    $\chi^{2}=\frac{\left(|r-s|-1\right)^{2}}{r+s}$

  3. This $\chi^{2}$ test statistic has 1 degree of freedom, so you get $p = P\left(X^{2}\ge\chi^{2}_{\text{df}=1}\right)$, as a measure of how unlikely you are to observe your test statistic if $H_{0}$ is true.

  4. Decide to reject or not if $p$ is less than or equal to your preferred type I error rate $\alpha$

  5. If you reject $H_0$ you conclude that you found evidence that $E$ is associated with $O$ (i.e. that $P(r)\ne P(s)$) at the $\alpha$ level of significance. If you fail to reject $H_0$ you conclude that you failed to find evidence that $E$ is associated with $O$ (i.e. failed to find that $P(r)\ne P(s)$).

Unpaired tests for association between binary variables
You have two choices that will get you to the same place: (1) a z test for difference in proportions, or (2) a 2-by-2 contingency table test.

z test for proportion difference between two groups (1 and 2, say the values of $E$; $p_{1}$ is $P\left(O=1|E=1\right)$, and $N_{1}$ is the sample size of group 1, and $p_{2}$ is $P\left(O=1|E=2\right)$, and $N_{2}$ is the sample size of group 2):

  1. Formulate your hypotheses:
    $H_{0}: p_{1} - p_{2} = 0$
    $H_{A}: p_{1} - p_{2} \ne 0$

  2. Calculate a test statistic from your data (the $1(1/N_{1} + 1/N_{2})/2$ and absolute value bits are the continuity correction by Yates):
    $z= \frac{|\hat{p}_{1}-\hat{p}_{2}|-\frac{\left(\frac{1}{N_{1}}+\frac{1}{N_{2}}\right)}{2}}{\sqrt{\hat{p}\left(1-\hat{p}\right)\left(\frac{1}{N_{1}}+\frac{1}{N_{2}}\right)}}$

    where $\hat{p}=\frac{\Sigma O=1}{N_{1}+N_{2}}$ (because if $H_{0}$ is true, then the best guess as to the true population probability of $O=1$ combines both groups into a single sample.

  3. This $z$ test statistic is distributed standard normal, so you get $p = P\left(|Z|\ge|z|\right)$, as a measure of how unlikely you are to observe your test statistic if $H_{0}$ is true.

  4. Decide to reject or not if $p$ is less than or equal to your preferred type I error rate $\alpha$

  5. If you reject $H_0$ you conclude that you found evidence that $P\left(O=1\right)$ is different depending on which group you are in at the $\alpha$ level of significance. If you fail to reject $H_0$ you conclude that you failed to find evidence that $P\left(O=1\right)$ is different depending on which group you are in at the $\alpha$ level of significance.

Instead you test the same unpaired data a different way using a 2-by-2 contingency table ($\chi^{2}$) test. However, you will organize your data as counts of individuals (not pairs) rather than proportions:

Observed counts of individuals
-------------------------------------------
O=1 & E=1 (a) |  O=1 & E=2 (b) | r1 = a + b
--------------+----------------+-----------
O=0 & E=1 (c) |  O=0 & E=2 (d) | r0 = c + d
--------------|----------------+-----------
  c1 = a + c  |    c2 = b + d  |  N=a+b+c+d
  1. Formulate your hypotheses (can use the same ones as the z test):
    $H_{0}: p_{1} - p_{2} = 0$
    $H_{A}: p_{1} - p_{2} \ne 0$

    If $H_{0}$ is true, then you would expect the counts in a, b, c, and d to be based on the marginal totals (same marginal totals as in the observed counts table):

    Expected counts of individuals
    --------------+----------------+-----------
       c1(r1/N)   |     c2(r1/N)   | r1 = a + b
    --------------+----------------+-----------
       c1(r0/N)   |     c2(r0/N)   | r0 = c + d
    --------------|----------------+-----------
      c1 = a + c  |    c2 = b + d  |  N=a+b+c+d
    
  2. Calculate a test statistic from your data (the "$-1/2$" and absolute value bits are the continuity correction by Yates):
    $\chi^{2}= \frac{\left(|a - c1(r1/N)|-\frac{1}{2}\right)^{2}}{c1(r1/N)} + \frac{\left(|b - c2(r1/N)|-\frac{1}{2}\right)^{2}}{c2(r1/N)} + \frac{\left(|c - c1(r0/N)|-\frac{1}{2}\right)^{2}}{c1(r0/N)} + \frac{\left(|d - c2(r0/N)|-\frac{1}{2}\right)^{2}}{c2(r0/N)}$

  3. This $\chi^{2}$ test statistic has 1 degree of freedom, so you get $p = P\left(X^{2}\ge\chi^{2}_{\text{df}=1}\right)$, as a measure of how unlikely you are to observe your test statistic if $H_{0}$ is true.

  4. Decide to reject or not if $p$ is less than or equal to your preferred type I error rate $\alpha$

  5. If you reject $H_0$ you conclude that you found evidence that $P\left(O=1\right)$ is different depending on which group you are in at the $\alpha$ level of significance. If you fail to reject $H_0$ you conclude that you failed to find evidence that $P\left(O=1\right)$ is different depending on which group you are in at the $\alpha$ level of significance.

Control variables You have two choices: (1) you can use an appropriate test as described above, and stratify your analysis on each level of a control variable(s), or you can switch to a regression format (there are several regressions for binary outcomes to choose from, the most common is, perhaps, logistic regression) in which you would fit models of the form:

$\text{logit}\left(O_{i}\right) = \beta_{0} + \beta_{E}E_{i} + \beta_{X}X_{i}$

where $X$ is a covariate, and the $\beta$s are interpreted as the natural log of the odds of $O$ ($\beta_{0}$ functioning as the intercept term for the logistic S-curve).

References
Edwards, A. L. (1948). Note on the “correction for continuity” in testing the significance of the difference between correlated proportions. Psychometrika, 13(3):185–187.

McNemar, Q. (1947). Note on the sampling error of the difference between two correlated proportions or percentages. Psychometrika, 12(2):153–157.

Yates, F. (1934). Contingency tables involving small numbers and the $\chi^{2}$ test. Supplement to the Journal of the Royal Statistical Society, 1(2):217–235.

$\endgroup$
1
$\begingroup$

Given that this is sale in an online webshop, your numbers in each cell in such a randomized* experiment are hopefully large (otherwise it may not be worthwhile to fine-tune minor details of the online shop), then I would not even bother with an exact test. E.g. a logistic regression (whether frequentist with likelihood ratio test + profile likelihood CI or Bayesian analysis with credible intervals) should be just fine and in fact allows you much more ability to adjust for additional information on customers (or the time of day they visit on etc.).

In fact, statistical significance may be a bit irrelevant. I.e. even things with a tiny impact will likely achieve statistical significance (and if they do not, the confidence/credible impact for the impact will likely exclude the possibility of a meaningful impact). When you have lots of possible things to potentially implement, then some of those with a tiny impact may not be a priority. Thus, it will likely be important to get an estimate of the effect of a change to the online shop, which again logistic regression could be a pretty good choice for.

*: Doing anything other than a randomized experiment, where you randomly pick visitors and show them one version of the webpage or the other is just making your life complicated for no reason. E.g. doing one thing in one month and then something else in the next month is just not a good idea (Suprise! Christmas presents sold better in December than in January!).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.