1
$\begingroup$

I have a dataset (structure explained below) for which I want to perform some tests on 2x2-tables.

The data is generated from a (randomized) experiment, and structured as follows:

I have n subjects, split into 2 treatment groups of equal size (each group has n/2 members; I observe each subject only within one of the treatments). Each of the subjects faces m trials, in each of which I get a response that can be coded as success/failure. (Note: in my case 2 < m << n.)

I am interested in testing whether there is a significant difference in success rate between groups.

If I had nm unrelated observations, I'd be happy performing a Fisher's exact test on the resulting 2x2 table (I am aware of the fact that Fisher's exact conditions on the margins, which is not actually appropriate in the given case - but this is not my issue here. The principal problem - as far as I understand - would apply to the basic versions of the common alternatives as well (including, e.g. a chi-sq., insofar they assume independence of observations.)

But the crux is that it seems unreasonable to assume that the repeated responses (by individual subjects) are independent: An uncorrected test should thus yield too low p-values.

Are there extensions to the typical non-parametric tests that allow to account for this "clustered" structure of my data?

(I found some references that mention the problem, but not a straightforward suggestion I which seems to aplly to my case - which might well be due to the fact that I am not sure exactly what to search for.

If possible, I'd like to retain individual trials as unit of observation, rather than per-subject frequency, as m is still small.

A test that is applicable to equal group sizes, and m constant across subjects would be a great start. Since I might go deeper in the analysis eventually, I'd also be grateful for any pointers how any suggested solution would be affected if these did not hold.)

Improve this question
$\endgroup$
5
  • $\begingroup$ Important details are missing here, but my first impression is that individual subjects should be 'scored' according their number of successes. That would indicate a two sample test (t or nonparametric, depending on nature of the data) rather than chi-sq or Fisher exact. // How many subjects? How many trials per subject? (Same number for each subject?) // Just from what you say I think it may be wrong to view individual trials as the 'unit of observation' and wish you would explain what you mean by that. From my reading of the question, seems clear subjects are the experimental units here. $\endgroup$
    – BruceET
    Commented Aug 1, 2018 at 21:07
  • $\begingroup$ Could you be more specific on which details you find missing? I'd be happy to fill them in. Concerning trials per subject, it's constant (=m) for the most important parts of analysis. (I might later restrict the subsample of trials analyzed for specific questions, thereby dropping this assumption - but an answer applying to constant m would already help me quite a bit, see last paragraph.) For the time being, n~160 (80 per group), the (constant) m ranges from 4 to 15 (depending on the exact question, analysis is on a different subsample of trials). $\endgroup$
    – DavidP
    Commented Aug 2, 2018 at 8:46
  • $\begingroup$ Concerning which tests are appropriate: Success ratio or count should be binomially distributed within subject (choosing a test that acknowledges this is what I meant by keeping trials as unit of observation, admittedly indeed a clumsy wording). I fail to see how a t-test or e.g. rank sum test on the success ratio or count is more appropriate than an accordingly modfied e.g. chi2/Fisher's (incorporating within-subject correlation), as the former disregard the binomial nature (and their distributional assumptions are in fact not fulfilled as far as I can see). $\endgroup$
    – DavidP
    Commented Aug 2, 2018 at 8:46
  • $\begingroup$ In the best case, using such a test would thus give up quite abit of power, I suspect? The most promising avenue (I believe) I have come across is an adjusted chi2 test, as proposed by Alan Donner: See stats.stackexchange.com/questions/50011/… for a reference. (In case it's of interest to anyone: a Stata implementation is cltest). $\endgroup$
    – DavidP
    Commented Aug 2, 2018 at 8:55
  • $\begingroup$ Hi @DavidP, it's been a while since there's been a reply to this post but I'll try anyway. Have you found a solution to your problem since? $\endgroup$
    – lulufofo
    Commented Mar 14 at 15:40

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.