# Fisher's Exact Test with weights?

Does anyone know of a variation of Fisher's Exact Test which takes weights into account? For instance sampling weights.
So instead of the usual 2x2 cross table, every data point has a "mass" or "size" value weighing the point.

Example data:

A B weight
N N 1
N N 3
Y N 1
Y N 2
N Y 6
N Y 7
Y Y 1
Y Y 2
Y Y 3
Y Y 4


Fisher's Exact Test then uses this 2x2 cross table:

A\B  N  Y All
N   2  2   4
Y   2  4   6
All  4  6  10


If we would take the weight as an 'actual' number of data points, this would result in:

A\B  N  Y All
N   4 13  17
Y   3 10  13
All  7 23  30


But that would result in much too high a confidence. One data point changing from N/Y to N/N would make a very large difference in the statistic.
Plus, it wouldn't work if any weight contained fractions.

I have a suspicion that 'exact' tests and sampling weights are essentially incompatible concepts. I checked in Stata, which has good facilities for sample surveys and reasonable ones for exact tests, and its 8 possible test statistics for a crosstab with sample weights don't include any 'exact' tests such as Fisher's.

The relevant Stata manual entry (for svy: tabulate twoway) advises using its default test in all cases. This default method is based on the usual Pearson's chi-squared statistic. To quote:

"To account for the survey design, the statistic is turned into an F statistic with noninteger degrees of freedom by using a second-order Rao and Scott (1981, 1984) correction".

Refs:

• Rao, J. N. K., and A. J. Scott. 1981. The analysis of categorical data from complex sample surveys: Chi-squared tests for goodness of fit and independence in two-way tables. Journal of the American Statistical Association 76:221–230.
• Rao, J. N. K., and A. J. Scott. 1984. On chi-squared tests for multiway contingency tables with cell proportions estimated from survey data. Annals of Statistics 12: 46–60.

Interesting question. What do you mean by weight?

I would be inclined to do a bootstrap...pick your favorite statistic (i.e. Fisher's Exact), and compute it on your data. Then assign new cells to each instance according to your null hypothesis, and repeat the process 999 times. This should give a pretty good empirical distribution for your test statistic under the null hypothesis, and allow easy computation of your p-value!

• Thanks! But I hoped for a statistic that is faster and more stable to calculate... Jul 29, 2010 at 15:25

One quick thing about sample weights - they are usually a way to incorporate some information about the population that one is sampling from - but usually they are based on "big sample" type scenarios (typically constrained BLUP or BLUE prediction in disguise). So I would imagine that sample weights will probably do no better than no weights. What would be better I think is to use the information about the population that the sample design was based on directly.

For example, on what basis were the selection probabilities calculated? My bet is that you knew a population total or some kind of population break-down which does not involve A or B (say age by sex groups). If this is not correct then I am about to waste some space, but if it is correct, and supposing you had population totals $R_{1},\dots,R_{k}$ for $k$ groups (or strata), and within each group you had a "mini" 2 by 2 contingency table. So we can now write $R_{1;11},R_{1;12},R_{1;21},R_{1;22},\dots$ as the "target" of our inference. Or perhaps it is the sum $\sum_{l=1}^{k}R_{l;ij}$ that is the target of inference (how many in the population give response N/N??). You are then trying to reason about $R_{l;ij}$ from the sampled numbers $r_{l;ij}$ subject to the constraint that $\sum_{i,j}R_{l;ij}=R_{l}$ for $(l=1,\dots,k)$. (maxent anyone?)

Note that if the sampling probabilities were based only on what data you were likely to receive, then they are irrelevant (and Fisher's exact test applies), because once you receive the data, you know what sample you received. So the coherent thing to do is to update the sampling probability to $P(D_{m})=1$ if the mth unit is in the sample, and $P(D_{m})=0$ if they weren't in the sample. However, usually the design is based in more information than just the data one is likely to observe. but note that it is the information rather than the survey design per se that is important. Design based inference is just a rather efficient way to incorporate all that information into your analysis.