A colleague has asked me to assess the data analytic approach taken by one of his students for a previous experiment. If you want to skip the details of the experimental design skip down to Data analysis Part 2 and My questions below.

Experimental design. In its simplest form, the experiment consists of a single subject completing a single session of 100 trials. In each session, the subject is first presented with a pre-testing set of stimuli that conform to a particular pattern. For the subsequent 100 trials, the subject is then presented with more stimuli and is asked (yes/no) whether they are consistent with the pre-test pattern. Although the stimuli were generated according to a fixed rule, the subject is not given the rule and therefore must infer and apply a rule based upon the observed pre-test pattern.

Hypotheses. The student has two goals for statistical analysis. The first is the obvious hypothesis test of whether the subject's performance is consistent with the actual stimuli-generating rule. The second set of hypotheses concern whether the subject's performance is also consistent with a variety of other possible rules hypothesized a priori by the experimenter. In particular, the student wants to know what the likelihoods of other possible rules are relative to the most likely rule for the subject.

Data analysis.

Part 1. The subject's data are treated as a random sample from a binomial process. The student assumed a 5% performance error rate for the subject, such that the probability of success (correct yes/no answer) for each trial is $θ = 0.95$.

For the first hypothesis, the student conducted a log-likelihood ratio test comparing the subject's observed performance ($\hat{θ}$) to the assumed population parameter ($θ$). The subject answered correctly on 93% ($k=93$) of the trials ($n=100$). The log-likelihood ratio is therefore:


The student cannot reject the null hypothesis ($H_0:\hat{θ}=θ)$ for $\alpha=0.05$ and therefore concludes that the subject's performance is consistent with the stimulus generating rule. This approach seems appropriate to me, although it may be questionable to assume each trial is i.i.d. within the session for a single subject. Nonetheless, I'm more concerned with the student's approach for the second set of hypotheses.

Part 2. The student is now interested in asking, given the subject's performance, what is the likelihood that the subject's performance is also consistent with a set of other possible rules? To answer this question, the student again fixes the population parameter, reflecting his expectation that the probability of answering correctly relative to any of the hypothesized rules is constant. That is to say, even if the stimulus set was generated according to rule A, the subject answering according to rule B is just as likely to select yes for any trials consistent with B (and vice versa).

In order to compare the rules, the student therefore had to recalculate the subject's $k$ for every hypothesized rule. For example, $k=20$ for rule B if the subject's performance on 20/100 trials generated with rule A was also consistent with B. Thus, the student generated a 'new' dataset for every rule, albeit assuming each to be drawn from the same underlying binomial process with distribution parameters $θ=0.95$ and $n=100$. The student then computes a ratio for every rule relative to the rule with the highest likelihood. For example, assume rule A has the highest success rate at $k=93$ and $k=89$ for rule B. The student would then calculate what he believed to be a relative likelihood ratio:

$$\exp([\ln\binom{100}{89}+(89)\ln(0.95)+(11)\ln(0.05)])- \\ \qquad\quad\;[\ln\binom{100}{93}+(93)\ln(0.95)+(7)\ln(0.05)]=0.068$$

My questions. Given that the population parameters are held constant, this is not a relative likelihood ratio but rather a probability ratio for $Pr(X=89)/Pr(X=93)$. Moreover, by holding the population parameters constant, my thinking is that there are zero degrees of freedom for the comparison. This would make the probability ratio meaningless for hypothesis testing with the $\chi^2$ distribution, as is done with the log-likelihood ratio.

Is my thinking here correct? If so, are there heuristic criteria or alternative approaches by which the magnitude of such a probability ratio could be interpreted? I've already thought of using simple $\chi^2$ goodness-of-fit or Fisher's exact tests for observed vs. expected yes/no responses, which would avoid these issues entirely and not require the 'generation' of new $k$ for every rule. Any other possible solutions would also be much appreciated. Thank you in advance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.