3
$\begingroup$

I've conducted a psychological experiment on the same subject, under two different condition. For each condition I've collected the number of correct and wrong answer for each stimulus (number of trials per stimulus = 10; number of stimulus = 15). From the data collected I fitted a psychometric curve (probit or logit curve), and now I want to compare the results.

Consider subject 1: His answers are collected in the following matrices

**COND_1**
    correct
cnt   Y  N
  1   0 10
  2   0 10
  3   0 10
  4   0 10
  5   0 10
  6   0 10
  7   1  9
  8   4  6
  9   5  5
  10  7  3
  11  8  2
  12  7  3
  13 10  0
  14 10  0
  15 10  0

**COND_2**
    correct
cnt   Y  N
  1   0 10
  2   2  8
  3   2  8
  4   3  7
  5   2  8
  6   4  6
  7   8  2
  8  10  0
  9   8  2
  10 10  0
  11  9  1
  12 10  0
  13 10  0
  14 10  0
  15 10  0

COND_1.1 <- matrix(c(0, 2, 2, 3, 2, 4, 8, 10, 8, 10, 9, 10, 10, 10, 10, 10, 8, 8, 7, 8, 6, 2, 0, 2, 0, 1, 0, 0, 0, 0), byrow=F, ncol=2) 
COND_1.2 <- matrix(c(0, 0, 0, 0, 0, 0, 1, 4, 5, 7, 8, 7, 10, 10, 10, 10, 10, 10, 10, 10, 10, 9, 6, 5, 3, 2, 3, 0, 0, 0), byrow=F, ncol=2)

cnt <- seq(from=0, to=1.4, by=0.1)

ddprob.1.1 <- glm(COND_1.1 ~ cnt, family = binomial(link = "probit")) 
ddprob.2.1 <- glm(COND_2.1 ~ cnt, family = binomial(link = "probit"))

enter image description here

Observing the plot, you can surely say that performance under condition 2 is better then the performance under condition 1. In fact if you compute some parameters like AUC, p25, p50, p75 (the value of the stimulus at which the number of correct answers are 25%, 50% and 75% each) [threshold parameters], you'll see that:

COND_1      COND_2
AUC    <    AUC
p25    >    p25
p50    >    p50
p75    >    p75

Now consider subject 2:

COND_1
    correct
cnt   Y  N
  1   0 10
  2   0 10
  3   0 10
  4   0 10
  5   3  7
  6   3  7
  7   4  6
  8   5  5
  9   9  1
  10  9  1
  11 10  0
  12 10  0
  13 10  0
  14 10  0
  15 10  0

COND_2
    correct
cnt   Y  N
  1   0 10
  2   1  9
  3   0 10
  4   4  6
  5   2  8
  6   6  4
  7   4  6
  8   7  3
  9   4  6
  10  7  3
  11  7  3
  12 10  0
  13  9  1
  14 10  0
  15 10  0


COND_1.2 <- matrix(c(0, 0, 0, 0, 3, 3, 4, 5, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 10, 7, 7, 6, 5, 1, 1, 0, 0, 0, 0, 0), byrow=F, ncol=2)
COND_2.2 <- matrix(c(0, 1, 0, 4, 2, 6, 4, 7, 4, 7, 7, 10, 9, 10, 10, 10, 9, 10, 6, 8, 4, 6, 3, 6, 3, 3, 0, 1, 0, 0), byrow=F, ncol=2)
cnt <- seq(from=0, to=1.4, by=0.1)

ddprob.1.1 <- glm(COND_1.2 ~ cnt, family = binomial(link = "probit"))
ddprob.2.1 <- glm(COND_2.2 ~ cnt, family = binomial(link = "probit"))

The plotted curves are shown in the image below.

enter image description here

Can you say what performance is better? Under condition 1 or under condition 2? Extrapolating AUC, p25, p50, p75, you have:

COND_1      COND_2
AUC     >   AUC
p25     >   p25
p50     =   p50
p75     <   p75

So my question is: Is there a method to say that a generic curve (logit, probit or whatelse) is HIGHER then another? Is there a method to compare (a single number of "performance") that describe the differences of the two curves? My example shows that AUC, p25, p50, p75 are not good parameters. I'd like to compute a single numeric parameter for each curves, to make a simple t-paired test, extrapolated from the distribution seen.

$\endgroup$
  • $\begingroup$ If in fact subject #2 performs better under Condition 1 for high values of the stimulus & better under Condition 2 for low values of the stimulus, do you really want a single number to sum up performance? $\endgroup$ – Scortchi Apr 13 '13 at 16:23
  • $\begingroup$ @Scortchi Maybe not (in my overall design, however, one value is sufficient), but if so, i could find a threshold of inversion of performances. But the new problem becomes: how to extract 2 parameters (one for low values of the stimulus, and one for high values) for describing performance with a statistical approach? (and not just "viewing" the plot) $\endgroup$ – Tommaso Apr 13 '13 at 16:29
  • 1
    $\begingroup$ Well I'd be thinking about using logistic regression with performance as the response, & with subject, condition, stimulus, & their interactions as the predictors. $\endgroup$ – Scortchi Apr 13 '13 at 16:32
  • $\begingroup$ How can one parameter possibly be sufficient? Or even two? And it's debatable whether those curves are proper: What is the "stimulus" and how does each stimulus vary? Your graph assumes that stimulus is interval level data (that is, that the difference between e.g. stimulus 2 and 3 is the same as between stimulus 4 and 5). $\endgroup$ – Peter Flom Apr 13 '13 at 19:22
  • $\begingroup$ @PeterFlom I've added some infos in the comment to the Scortchi answer. Yes, the stimulus is continuous, and yes the interval level data is constant. In the first graph it appears to change the intercept, while in the second graph it appears to change the slope. How can the overall performance of SUBJ-n be described properly? $\endgroup$ – Tommaso Apr 15 '13 at 10:38
1
$\begingroup$

In short: if which condition betters performance depends on the stimulus level, you can't say which condition betters performance overall without some kind of averaging over stimulus levels; which is only likely to be a meaningful procedure if you have data on the distribution of stimulus levels over some population or other. And surely ignoring what's interesting in your results just to be able to do a boring old t-test is putting the cart before the horse.

Logistic regression could be really good here. Assuming the stimulus can sensibly be treated as a continuous variable (see @Peter's comment), try fitting

$$\log\frac{\pi}{1-\pi}=\beta_0 +\beta_p p + \beta_c c + \beta_s s + \beta_{pc} pc + \beta_{ps} ps + \beta_{cs} cs$$

where $\pi$ is the probability of success, $p$ is a dummy variable for person, $c$ is a dummy variable for stimulus, & $s$ is the strength of the stimulus. Each of the estimated coefficients (the $\beta$s) tells you something useful, & you can use a likelihood ratio test instead of a t-test.

$\endgroup$
  • $\begingroup$ First of all, thanks for the answer. Some explanation: stimulus is continuous (i.e. the luminance of a monitor, from dark to best max luminance); conditions are: watch the monitor with your right eye (COND_1) or your left eye (COND_2); the question for each trial is: "the color you see on the monitor is blue or green?". I hope thiese new infos are meaningful. My final question was so: is better the right or the left eye to detect colors? This is the reason why I was trying a single parameter to describe how much one eye is better (if so) then the other. $\endgroup$ – Tommaso Apr 14 '13 at 9:06
  • $\begingroup$ I don't know if the interaction is physiologically plausible. You can test the significance of the $cs$ term. It could suggest there's something wrong with the experimental protocol - sequence effects perhaps? $\endgroup$ – Scortchi Apr 14 '13 at 11:05
  • $\begingroup$ And by the way, logistic regression is equivalent to describing each curve with two parameters - the intercept & slope of the fitted logit. $\endgroup$ – Scortchi Apr 14 '13 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.