1
$\begingroup$

I am concerned about the way "response" to certain experimental manipulations in psychology is defined and then used to claim that a specific subject did or did not "respond" to justify exclusion of "non-reponders" from the statistical analysis.

More specifically, I'm talking about studies testing both Pavlovian fear conditioning and fear extinction in humans. During conditioning, one stimulus (CS+) is repeatedly paired with an unconditioned stimulus (US) while another stimulus (CS-) is never paired with the US. Conditioning across subjects is found when the average response to the CS+ is significantly higher compared to the CS-. However, many studies exclude subjects where conditioning "did not work" by choosing arbitrary thresholds to make that claim. For instance, if a particular subject's average response to the CS+ was not higher (or not higher than 0.2 units or whatever arbitrary threshold) compared to the CS-, these studies might label the subject a "non-responder" or "non-learner" and exclude it. The thresholds that are applied to make this distinction vary from study to study.

I suspect that many studies do this to exploit the "researchers degrees of freedom", i.e., to increase their chances of finding a significant result. However, I assume there is a statistical argument as well that it doesn't make sense to exclude "non-responders". Stephen Senn has repeatedly made the point for clinical trials that by design they usually do not allow for such a distinction, i.e., it is impossible to tell in any individual case whether the subject responded to the drug or not.

I'm wondering if a similar argument can be made for fear learning experiments in psychology as well, that naively using response-rates (to manipulations) as an indicator of variation in true response, and my question is thus: am I right that one usually cannot decide whether a particular subject did in fact respond to a particular manipulation (in this case, fear conditioning)?

Edit: This would be a (randomly picked) example: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3501228/. It says that "we adopted selection criteria to establish that (1), conditioning was successful (defined as last two trials of acquisition > for both the CS + s vs. the CS−) and (2), extinction was successful (defined as the first two trials of extinction (trial 1–2) > the last two trials of extinction (trial 11–12). Adopting these selection criteria resulted in the exclusion of five subjects".

Improve this question
$\endgroup$
5
  • $\begingroup$ I glanced at Golkar et al. and it seems that the analyses focus on within-subjects effects; in fact, there are no between-subjects manipulations. Since no a priori rationale is provided for the particular exclusion criteria, there may well have been abuse of researcher degrees of freedom. (This is just the same as for any exclusion criteria.) But, I don't think Senn's point applies since basic conditioning effects are supposed to happen to all humans, and there's no between-subjects manipulation. $\endgroup$
    – Kodiologist
    Commented Oct 7, 2016 at 23:57
  • $\begingroup$ Oh no, wait: "Adopting these selection criteria resulted in the exclusion of five subjects, but did not alter our reported findings." So no, no possible abuse there. $\endgroup$
    – Kodiologist
    Commented Oct 7, 2016 at 23:58
  • $\begingroup$ But isn't this the very problem: to assume that an effect will happen to all humans, i.e., instead of thinking about effects in terms of sizes and variations? I mean, sure, associative learning is presumably a pretty large and well replicated effect; but is it large enough to assume it must be present in each individual case? I doubt it; in fact, you will find even fairly recent studies that fail to find conditioning effects in whole samples healthy humans. Of course, this may depend on measurement error and might thus well depend on the way you operationalize conditioning. $\endgroup$
    – tura
    Commented Oct 8, 2016 at 16:23
  • $\begingroup$ Behaviorists, at least, tend to take it as a basic assumption that classical conditioning and instrumental conditioning are universal processes in animals, with any individuals who lose one of these fundamental abilities quickly dying due to failure to learn about its environment. Are they wrong about that? Possibly. I don't know. $\endgroup$
    – Kodiologist
    Commented Oct 8, 2016 at 16:32
  • $\begingroup$ It might well be an universal process in animals. But that doesn't preclude that there might be (and will be) variation between individuals. For some it just might take longer than, let's say, 8 trials. And I'm saying there's no reason to believe that those individuals qualify as non-learners or whatever and should be excluded from an experiment. $\endgroup$
    – tura
    Commented Oct 8, 2016 at 18:05

1 Answer 1

Reset to default
1
$\begingroup$

Although the Golkar et al study stated, "did not alter our reported findings" (as quoted by Kodiologist), the notion that excluding 5 subjects did not change results is improbable. If those 5 had been included, the significance p-value would be increased, certainly. And, averages and other statistical numbers would differ.
So, exclusions have an effect. I think, though, this is not in any way a scandal. As readers and reviewers and researchers, we need to understand how to interpret what we read. In cases involving exclusion, we might infer something like, "When this process is applicable, it succeeds...(insert reported results here.)"

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.