4
$\begingroup$

Experimental Design

4 Animals repeated a binary choice task with 60 trials over 15 days. For each day, we calculated the proportion for which the animal chose the given 'correct' stimulus. In addition to this choice data, we have a single continuous behavioral measure for each animal (4 data points). We wish to relate this behavioral measure with the animals' overall proportion of 'correct' choices. You can think of this variable as a characteristic of each animal, such as its sex or gene variant, etc.

Initial Analysis

We initially calculated the mean proportion for each animal across all days and calculated the Pearson Correlation of the mean proportion with the behavioral measure (-.99 [-.999,-.3160], 95% CI); however, a reviewer commented on how this was meaningless as we have such a small N.

Question: What other method(s) should we utilize to demonstrate this relationship?


Data

As requested by whuber, the data is as follows:

 means   behavior
 0.672         13
 0.637         21
 0.596         23
 0.513         39

As per my paranoia, the numbers are changed quite minimally. This should not alter any of the interpretations.

$\endgroup$
4
  • $\begingroup$ More information is needed if you want good answers. E.g., suppose the total numbers of correct choices made were $0,1,3,4$ for animals with behavior scores of $1,.9,.15,0$ respectively (which is consistent with your description: $\rho=-0.998$, CI=$[-0.9996,-0.41]$). These counts have Binomial distributions; assuming the observations reflect the true chances, there's less than a $1$% chance of a positive correlation. But if the numbers were $30,31,33,34$--equally consistent with your description--there is now a $21$% chance of a positive correlation. $\endgroup$
    – whuber
    Commented Jun 10, 2013 at 14:28
  • $\begingroup$ That is very interesting to me. Can you point me to where I can find more information on the relation between binomial count and correlation? The counts of correct answers for each animal range from about 300 to 700, but I don't recall ever learning anything about the magnitude of count's effect on correlation. $\endgroup$
    – 123456
    Commented Jun 10, 2013 at 15:35
  • $\begingroup$ With that range you're probably ok, but it would be worth looking more closely. (The idea is that the variance of the counts is approximately proportional to the counts themselves, so when the spacing between the counts is of the same order of magnitude as their square roots, you cannot reliably order the counts.) Since your data amount to just four pairs of numbers, perhaps you could post them? There could also be concerns about the "behavioral measure": how is it assessed and how accurate and reliable is it? $\endgroup$
    – whuber
    Commented Jun 10, 2013 at 16:14
  • $\begingroup$ Ah, that makes sense. I don't have the data on hand, but I agree with your point about the behavioral measure; the reviewers did mention some concerns. Thanks for the assistance--when I get a hold of the data I'll follow up. $\endgroup$
    – 123456
    Commented Jun 10, 2013 at 16:38

1 Answer 1

3
$\begingroup$

The reviewer is incorrect. It's not very generalizable or strong evidence but it's also not entirely meaningless. The reviewer may be referring to the fact that the distribution of correlations is nearly flat when the N is 4 and the true correlation is 0. Any correlation is about equally likely as any other. So, from that standpoint it looks meaningless. But if you've got an argument that your correlation should be negative from the literature or logic then you're not on all that shaky a ground. Furthermore, if you're arguing against a theory with a positive correlation you have a pretty strong set of data. It depends alot on what you're arguing.

Make sure that your conclusions are adequately circumspect in recognition of the weaknesses here.

The method you've used to calculate your CI suggests r > -0.30 is unlikely. If the reviewer doesn't like the method you used to assess that then let them argue that. There are better and worse methods for getting the CI but they can't just ignore your statistics and ignore the magnitude of the effect. You might consider that the achieved power here is 0.79 (test against 0 using a Fisher's Z transform method). That's relatively good for behavioural science. But keep in mind that power estimate assumes you've got a good estimate of the true value. Very likely you've over estimated. In that case the power gets substantially lower very quickly. At a correlation of -0.95 the power is only 0.5.

$\endgroup$
1
  • $\begingroup$ Great--thanks for the advice. I'll discuss this with my colleagues and see where we can take it from here. It's nice to see that the reviewer's harsh response wasn't completely correct! $\endgroup$
    – 123456
    Commented Jun 10, 2013 at 4:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.