Best way to calculate "distance" between two segments expressing study participants' probability estimations

Question

I am working on a task in which participants estimate the probability that a series of beads are extracted from one of two hidden jars. The beads are extracted one by one, with replacement.

The two jars contain beads in two colors, yellow and black, in different proportions. Jar A contains 85% yellow beads and 15% black beads, and jar B contains 85% black beads and 15% yellow beads.

While the jars are hidden to the participant, he or she is aware of the difference between them - therefore he or she can estimate the probability that a sequence of beads is extracted specifically from one of the two jars.

General Example

After each extraction, the participant always answers the same question: "What's the probability that the sequence was extracted from jar A?

Then, if the first bead extracted is yellow, it's more likely that bead was extracted from jar A; if the second bead is also yellow, it's now even more likely that the sequence of beads was extracted from jar A, and so on.

The estimation, of course, changes as the participant is shown more beads. All participants are shown the same sequence.

Actual Task

At "Event 0," the participant is asked the question before seeing any beads. This is why the estimation is at .5.

Then, at Event 1, the participant is shown a whole sequence of beads that have already been extracted. In this case, 8 yellow beads and 2 black beads.

That's why, at this point (Event 1), the sequence is most likely coming from jar A.

After that, the participant is shown more beads, now extracted one by one. They all happen to be black (the participant does not know this beforehand, of course.) That's why the probability estimation slowly decreases.

Final sequence:

You can see below, the red line, as the probability estimations from one participant. On the x-axis, you can see the extraction number, on the y-axis the probability estimation. In black you can see the ideal observer's estimation, that is, the correct probabilities.

Ideal Observer

This is how I calculated the optimal probabilities.

At the very start (Event 0, before any draw) we have:

Priors:

$P(A)= 0.5$ (probability of jar being A)

$P(B)= 0.5$ (probability of jar being B)

We also know that:

$P(yellow|A) = .85$

$P(yellow|B) = .15$

The Bernoulli formula is:

$P(X|A) = P(yellow|A)^k P(yellow|B)^{n-k}$

$P(X|B) = P(black|A)^{n-k} P(black|B)^k$

We start with a sequence with $n=10$ and $k=8$:

$P(X|A) = .85^8 \cdot .15^2 = .272 \cdot .023 = .006$ $P(X|B) = .85^2 \cdot .15^8 = .722 \cdot 2.56E-7 = 1.85E-7$

Then the posterior is:

$$ P(A|X) = \frac{P(A) P(X|A)} {P(A) P(X|A) + P(B) P(X|B)} = \frac{.5 \cdot.006}{.5 \cdot.006 + .5 \cdot1.85E-7} = 0.9999 $$

Now for Event 2 we have another draw, that's a black bead, so we have 8 yellow beads and 3 black beads:

$P(X|A) = .85^8 \cdot .15^3 = .272 \cdot .0034 = .0009$ $P(X|B) = .85^3 \cdot .15^8 = .614 \cdot 2.56e-7 = .0014$

Then the posterior is:

$$ P(A|X) = \frac{P(A) P(X|A)} {P(A) P(X|A) + P(B) P(X|B)} = \frac{.5 \cdot.0009}{.5 \cdot.0009 + .5 \cdot 1.57E-07} = 0.9998 $$

R Plot

library(ggplot2)
library(scales)

# participant's probability estimations

participant <- structure(list(event = c(0, 1, 2, 3, 4, 5, 6, 7, 8, 9), prob_est = c(0.46, 
 0.98, 0.89, 0.72, 0.53, 0.21, 0.24, 0.12, 0.09, 0.01)), class = "data.frame", row.names = c("1", 
"2", "3", "4", "5", "6", "7", "8", "9", "10"))

# ideal observer's probability estimations

ideal_observer <- structure(list(event = c(0, 1, 2, 3, 4, 5, 6, 7, 8, 9), prob = c(0.5, 
0.99996979903057, 0.999828885289768, 0.999031123657329, 0.994534412955466, 
0.969798657718121, 0.85, 0.5, 0.15, 0.0302013422818792)), row.names = c(NA, 
10L), class = "data.frame")

plot <- ggplot(data=subset(participant, event<=9), aes(x = event, y = prob_est, col="red"))  + 
        geom_point(cex=1.5)+
        geom_line(lwd=0.9)+
        labs(x="Event Number", y="Probability") + 
        scale_y_continuous(breaks=pretty_breaks(n=10), limits = c(0,1))+
        scale_x_continuous(breaks=pretty_breaks(n=10))+
        geom_line(data=subset(ideal_observer, event<=9), aes(x = event, y = prob),col="black",lwd=0.9)+
        geom_point(data=subset(ideal_observer, event<=9), aes(x = event, y = prob),col="black",cex=1.5)
plot

The problem

I would like to define a "profile" for each participant, based on how he or she responds to the task. Basically, the end goal is to assess each partcipant's performance, so that I can then correlated them with psychometric measures.

The partcipant performance is defined with respect to how much he or she is far off from the ideal observer. I thought I could just calculate the distance between each pair of points on the y-axis and sum them up.

# calculating discrepancy from ideal performance
difference <-  sum(participant[,2] - ideal_observer[,2])
difference
#> [1] -2.743364

^{Created on 2021-03-24 by the reprex package (v0.3.0)}

Question

Does this make sense? I was wondering if there are better ways to perform this type of analysis. Is there a way I can retain more information about the participant's choices? For example, I thought I could fit one curve to the participant's response and one curve to the ideal observer's estimation and evaluate the difference between the parameters defining the curves, but I am not sure about how to go about that.

Note

I ended up doing something very simple (see also here and here to build on that.)

Answer 1

Provided the distribution must originate from either Jar-A (p=0.85) or Jar-B (p=0.15), the probability that a particular outcome originates from Jar-A should be equivalent to the proportion of probabilities to observe the outcome at given p=0.15 vs p=0.85.

With this approach, the probabilities would be:

[0.5, 0.85, 0.9697986577181208, 0.9945344129554655, 0.9697986577181208, 0.9945344129554656, 0.9990311236573288, 0.9998288852897683, 0.9999697990305695, 0.9998288852897683, 0.9999697990305696, 0.9998288852897682, 0.9990311236573287, 0.9945344129554656, 0.9697986577181208, 0.8500000000000002, 0.5000000000000003, 0.15000000000000022, 0.03020134228187925, 0.005465587044534424, 0.0009688763426712303]

Now to your actual question of how to evaluate the quality of a participants predictions: Please be aware that sum(vector1 - vector2) is equivalent to sum(vector1) - sum(vector2), which quite certainly is not what you want. To your question of fitting the participants answers to a curve, I would certainly not do that, as it seems like unnecessary data-manipulation.

Some obvious options, besides the sum of absolute differences along all points, include normalizations like "L2 norm" or "L 0.5 norm", the sum of ranked differences or simply the median difference among the points or the sum of distances weighted by the inverse standard deviation among participants (plus offset) at the point..