I am working on a task in which participants estimate the probability that a series of beads are extracted from one of two hidden jars. The beads are extracted one by one, with replacement.

The two jars contain beads in two colors, yellow and black, in different proportions. Jar A contains 85% yellow beads and 15% black beads, and jar B contains 85% black beads and 15% yellow beads.

enter image description here

While the jars are hidden to the participant, he or she is aware of the difference between them - therefore he or she can estimate the probability that a sequence of beads is extracted specifically from one of the two jars.

General Example

After each extraction, the participant always answers the same question: "What's the probability that the sequence was extracted from jar A?

Then, if the first bead extracted is yellow, it's more likely that bead was extracted from jar A; if the second bead is also yellow, it's now even more likely that the sequence of beads was extracted from jar A, and so on.

The estimation, of course, changes as the participant is shown more beads. All participants are shown the same sequence.

Actual Task

At "Event 0," the participant is asked the question before seeing any beads. This is why the estimation is at .5.

Then, at Event 1, the participant is shown a whole sequence of beads that have already been extracted. In this case, 8 yellow beads and 2 black beads.

enter image description here

That's why, at this point (Event 1), the sequence is most likely coming from jar A.

After that, the participant is shown more beads, now extracted one by one. They all happen to be black (the participant does not know this beforehand, of course.) That's why the probability estimation slowly decreases.

Final sequence:

enter image description here

You can see below, the red line, as the probability estimations from one participant. On the x-axis, you can see the extraction number, on the y-axis the probability estimation. In black you can see the ideal observer's estimation, that is, the correct probabilities.

Ideal Observer

This is how I calculated the optimal probabilities.

At the very start (Event 0, before any draw) we have:


$P(A)= 0.5$ (probability of jar being A)

$P(B)= 0.5$ (probability of jar being B)

We also know that:

$P(yellow|A) = .85$

$P(yellow|B) = .15$

The Bernoulli formula is:

$P(X|A) = P(yellow|A)^k P(yellow|B)^{n-k}$

$P(X|B) = P(black|A)^{n-k} P(black|B)^k$

We start with a sequence with $n=10$ and $k=8$:

$P(X|A) = .85^8 \cdot .15^2 = .272 \cdot .023 = .006$ $P(X|B) = .85^2 \cdot .15^8 = .722 \cdot 2.56E-7 = 1.85E-7$

Then the posterior is:

$$ P(A|X) = \frac{P(A) P(X|A)} {P(A) P(X|A) + P(B) P(X|B)} = \frac{.5 \cdot.006}{.5 \cdot.006 + .5 \cdot1.85E-7} = 0.9999 $$

Now for Event 2 we have another draw, that's a black bead, so we have 8 yellow beads and 3 black beads:

$P(X|A) = .85^8 \cdot .15^3 = .272 \cdot .0034 = .0009$ $P(X|B) = .85^3 \cdot .15^8 = .614 \cdot 2.56e-7 = .0014$

Then the posterior is:

$$ P(A|X) = \frac{P(A) P(X|A)} {P(A) P(X|A) + P(B) P(X|B)} = \frac{.5 \cdot.0009}{.5 \cdot.0009 + .5 \cdot 1.57E-07} = 0.9998 $$

R Plot


# participant's probability estimations

participant <- structure(list(event = c(0, 1, 2, 3, 4, 5, 6, 7, 8, 9), prob_est = c(0.46, 
 0.98, 0.89, 0.72, 0.53, 0.21, 0.24, 0.12, 0.09, 0.01)), class = "data.frame", row.names = c("1", 
"2", "3", "4", "5", "6", "7", "8", "9", "10"))

# ideal observer's probability estimations

ideal_observer <- structure(list(event = c(0, 1, 2, 3, 4, 5, 6, 7, 8, 9), prob = c(0.5, 
0.99996979903057, 0.999828885289768, 0.999031123657329, 0.994534412955466, 
0.969798657718121, 0.85, 0.5, 0.15, 0.0302013422818792)), row.names = c(NA, 
10L), class = "data.frame")

plot <- ggplot(data=subset(participant, event<=9), aes(x = event, y = prob_est, col="red"))  + 
        labs(x="Event Number", y="Probability") + 
        scale_y_continuous(breaks=pretty_breaks(n=10), limits = c(0,1))+
        geom_line(data=subset(ideal_observer, event<=9), aes(x = event, y = prob),col="black",lwd=0.9)+
        geom_point(data=subset(ideal_observer, event<=9), aes(x = event, y = prob),col="black",cex=1.5)

The problem

I would like to define a "profile" for each participant, based on how he or she responds to the task. Basically, the end goal is to assess each partcipant's performance, so that I can then correlated them with psychometric measures.

The partcipant performance is defined with respect to how much he or she is far off from the ideal observer. I thought I could just calculate the distance between each pair of points on the y-axis and sum them up.

# calculating discrepancy from ideal performance
difference <-  sum(participant[,2] - ideal_observer[,2])
#> [1] -2.743364

Created on 2021-03-24 by the reprex package (v0.3.0)


Does this make sense? I was wondering if there are better ways to perform this type of analysis. Is there a way I can retain more information about the participant's choices? For example, I thought I could fit one curve to the participant's response and one curve to the ideal observer's estimation and evaluate the difference between the parameters defining the curves, but I am not sure about how to go about that.


I ended up doing something very simple (see also here and here to build on that.)

  • 1
    $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ Commented Mar 26, 2021 at 10:04

1 Answer 1


Provided the distribution must originate from either Jar-A (p=0.85) or Jar-B (p=0.15), the probability that a particular outcome originates from Jar-A should be equivalent to the proportion of probabilities to observe the outcome at given p=0.15 vs p=0.85.

enter image description here

With this approach, the probabilities would be:

[0.5, 0.85, 0.9697986577181208, 0.9945344129554655, 0.9697986577181208, 0.9945344129554656, 0.9990311236573288, 0.9998288852897683, 0.9999697990305695, 0.9998288852897683, 0.9999697990305696, 0.9998288852897682, 0.9990311236573287, 0.9945344129554656, 0.9697986577181208, 0.8500000000000002, 0.5000000000000003, 0.15000000000000022, 0.03020134228187925, 0.005465587044534424, 0.0009688763426712303]

enter image description here

Now to your actual question of how to evaluate the quality of a participants predictions: Please be aware that sum(vector1 - vector2) is equivalent to sum(vector1) - sum(vector2), which quite certainly is not what you want. To your question of fitting the participants answers to a curve, I would certainly not do that, as it seems like unnecessary data-manipulation.

Some obvious options, besides the sum of absolute differences along all points, include normalizations like "L2 norm" or "L 0.5 norm", the sum of ranked differences or simply the median difference among the points or the sum of distances weighted by the inverse standard deviation among participants (plus offset) at the point..

  • 1
    $\begingroup$ See my edit. (evaluation of distance) $\endgroup$
    – KaPy3141
    Commented Mar 24, 2021 at 23:59
  • 1
    $\begingroup$ @Emy. It's a vector because you are forgetting the second sum() function. $\endgroup$
    – KaPy3141
    Commented Mar 25, 2021 at 20:55
  • 1
    $\begingroup$ Thank you, you are right about the vector. $\endgroup$
    – Emy
    Commented Mar 25, 2021 at 20:59
  • 1
    $\begingroup$ Thank you for your ideas about the distance calculations. I ended up going for something simple that I could use as a starting point to build on with a more refined analysis. I am curious about what you mean by "L2 norm or L 0.5 norm" and by weighting by the inverse sd (plus offset) - can you point me to some examples? $\endgroup$
    – Emy
    Commented Mar 25, 2021 at 21:05
  • 1
    $\begingroup$ For L2 norm (applied this would be the root of suqared differences) and other norms see here: en.m.wikipedia.org/wiki/Norm_(mathematics) With "0.5 norm" I wanted to highlight the possibility to use exponents smaller 1 to increase the weight of precise guesses. By weighting by inverse standard deviation you would count points stronger, that are difficult to guess, with low consensus among participants, the offset is needed to avoid infinite weight at standard deviation of 0. (division by 0) $\endgroup$
    – KaPy3141
    Commented Mar 25, 2021 at 21:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.