I am working on a task in which participants estimate the probability that a series of beads are extracted from one of two hidden jars. The beads are extracted one by one, with replacement.
The two jars contain beads in two colors, yellow and black, in different proportions. Jar A contains 85% yellow beads and 15% black beads, and jar B contains 85% black beads and 15% yellow beads.
While the jars are hidden to the participant, he or she is aware of the difference between them - therefore he or she can estimate the probability that a sequence of beads is extracted specifically from one of the two jars.
General Example
After each extraction, the participant always answers the same question: "What's the probability that the sequence was extracted from jar A?
Then, if the first bead extracted is yellow, it's more likely that bead was extracted from jar A; if the second bead is also yellow, it's now even more likely that the sequence of beads was extracted from jar A, and so on.
The estimation, of course, changes as the participant is shown more beads. All participants are shown the same sequence.
Actual Task
At "Event 0," the participant is asked the question before seeing any beads. This is why the estimation is at .5.
Then, at Event 1, the participant is shown a whole sequence of beads that have already been extracted. In this case, 8 yellow beads and 2 black beads.
That's why, at this point (Event 1), the sequence is most likely coming from jar A.
After that, the participant is shown more beads, now extracted one by one. They all happen to be black (the participant does not know this beforehand, of course.) That's why the probability estimation slowly decreases.
Final sequence:
You can see below, the red line, as the probability estimations from one participant. On the x-axis, you can see the extraction number, on the y-axis the probability estimation. In black you can see the ideal observer's estimation, that is, the correct probabilities.
Ideal Observer
This is how I calculated the optimal probabilities.
At the very start (Event 0, before any draw) we have:
Priors:
$P(A)= 0.5$ (probability of jar being A)
$P(B)= 0.5$ (probability of jar being B)
We also know that:
$P(yellow|A) = .85$
$P(yellow|B) = .15$
The Bernoulli formula is:
$P(X|A) = P(yellow|A)^k P(yellow|B)^{n-k}$
$P(X|B) = P(black|A)^{n-k} P(black|B)^k$
We start with a sequence with $n=10$ and $k=8$:
$P(X|A) = .85^8 \cdot .15^2 = .272 \cdot .023 = .006$ $P(X|B) = .85^2 \cdot .15^8 = .722 \cdot 2.56E-7 = 1.85E-7$
Then the posterior is:
$$ P(A|X) = \frac{P(A) P(X|A)} {P(A) P(X|A) + P(B) P(X|B)} = \frac{.5 \cdot.006}{.5 \cdot.006 + .5 \cdot1.85E-7} = 0.9999 $$
Now for Event 2 we have another draw, that's a black bead, so we have 8 yellow beads and 3 black beads:
$P(X|A) = .85^8 \cdot .15^3 = .272 \cdot .0034 = .0009$ $P(X|B) = .85^3 \cdot .15^8 = .614 \cdot 2.56e-7 = .0014$
Then the posterior is:
$$ P(A|X) = \frac{P(A) P(X|A)} {P(A) P(X|A) + P(B) P(X|B)} = \frac{.5 \cdot.0009}{.5 \cdot.0009 + .5 \cdot 1.57E-07} = 0.9998 $$
R Plot
library(ggplot2)
library(scales)
# participant's probability estimations
participant <- structure(list(event = c(0, 1, 2, 3, 4, 5, 6, 7, 8, 9), prob_est = c(0.46,
0.98, 0.89, 0.72, 0.53, 0.21, 0.24, 0.12, 0.09, 0.01)), class = "data.frame", row.names = c("1",
"2", "3", "4", "5", "6", "7", "8", "9", "10"))
# ideal observer's probability estimations
ideal_observer <- structure(list(event = c(0, 1, 2, 3, 4, 5, 6, 7, 8, 9), prob = c(0.5,
0.99996979903057, 0.999828885289768, 0.999031123657329, 0.994534412955466,
0.969798657718121, 0.85, 0.5, 0.15, 0.0302013422818792)), row.names = c(NA,
10L), class = "data.frame")
plot <- ggplot(data=subset(participant, event<=9), aes(x = event, y = prob_est, col="red")) +
geom_point(cex=1.5)+
geom_line(lwd=0.9)+
labs(x="Event Number", y="Probability") +
scale_y_continuous(breaks=pretty_breaks(n=10), limits = c(0,1))+
scale_x_continuous(breaks=pretty_breaks(n=10))+
geom_line(data=subset(ideal_observer, event<=9), aes(x = event, y = prob),col="black",lwd=0.9)+
geom_point(data=subset(ideal_observer, event<=9), aes(x = event, y = prob),col="black",cex=1.5)
plot
The problem
I would like to define a "profile" for each participant, based on how he or she responds to the task. Basically, the end goal is to assess each partcipant's performance, so that I can then correlated them with psychometric measures.
The partcipant performance is defined with respect to how much he or she is far off from the ideal observer. I thought I could just calculate the distance between each pair of points on the y-axis and sum them up.
# calculating discrepancy from ideal performance
difference <- sum(participant[,2] - ideal_observer[,2])
difference
#> [1] -2.743364
Created on 2021-03-24 by the reprex package (v0.3.0)
Question
Does this make sense? I was wondering if there are better ways to perform this type of analysis. Is there a way I can retain more information about the participant's choices? For example, I thought I could fit one curve to the participant's response and one curve to the ideal observer's estimation and evaluate the difference between the parameters defining the curves, but I am not sure about how to go about that.
Note
I ended up doing something very simple (see also here and here to build on that.)