6
$\begingroup$

Say someone who is well practiced (appears to have reached a performance plateau) shoots 20 free throws on 15 different days and is successful the number of times shown in the upper histogram (dat in the code).

  1. My understanding is that the distribution of outcomes should be predicted by the binomial distribution. Is this correct?

  2. The expected variance is $np(1-p)$, where $n = 20$ (the number of trials per session) and $p = {\rm mean}/n =.65$ or the average percent of successes.

  3. I could not figure out how to theoretically calculate the distribution of sample variances, so ran a Monte Carlo simulation. These results are shown in the lower panel. the mean of these variances matches with the theoretical expected variance, but the variance of the data is much less.

R code:

dat  <- c(12,12,13,12,13,12,12,14,13,13,14,13,14,13,14)
n    <- 20
p    <- mean(dat)/n
Nobs <- length(dat)

sim.vars = matrix(nrow=10000)
for(s in 1:10000){
  sim.vars[s] <- var(rbinom(Nobs, n, p))
}
par(mfrow=c(2,1))
hist(dat,      breaks=seq(0,20,by=1))
hist(sim.vars, breaks=20)


> var(dat)       # Variance of Data
[1] 0.6380952
> n*p*(1-p)      # Expected Variance given binomial model
[1] 4.569778
> mean(sim.vars) # Mean of simulated sample variances
[1] 4.542159

enter image description here

@Whuber, I hit enter when the cursor was outside the text box and it submitted before completing the question. I apologize. The first thing I wanted to know if I have made some error anywhere in my thinking (choice of binomial model, simulation, calculation), which your comment suggests I have not.

The second is what processes could possibly generate such data? I have >30 like this from multiple sources, so it is probably not data entry error or made up data. The actual task is not shooting free throws but you can take my word that it really is an equivalent situation.

This peculiarity of the data has not been noted previously. Others have interpreted such data as representing the max performance level achieved, and compared group averages under different conditions. Difference between individuals have been interpreted as differences in skill level, somehow related to neurological characteristics. As far as I can tell, this interpretation (as plateau/ asymptote/ max performance) implies sampling from a binomial distribution, which is really inconsistent with the underdispersion.

An analogous situation would be someone flipping a coin 20 times and always getting 9/10/11 heads. This is too consistent. The only mechanism I have thought of is introducing negative correlation between consecutive trials. Something like:

if(dat[t-1]=success){ p=0 }else{ p=0.95 } # Arbitrary probs used for example
dat[t]=sample(c(miss,success),1,prob=c(1-p,p))

What other processes could result in this underdispersion? The literature on underdispersion appears to be very sparse. I found it consists mostly of simply finding distributions that can fit such data that lack any clear physical interpretation. That type of analysis is not of interest to me here. Perhaps I missed something due to using inappropriate terminology?

Edit2: @whuber In response to your second comment: It really is just like the free throws, almost any explanation that works for that will also apply. An exception is that a person may purposefully miss on the free throw task to maintain a certain score, while that is implausible here.

The task requires motor coordination to attain a goal. A success requires performing a sequence of movements in the correct order, each in the correct fashion (of course with some level of variation). There may also be multiple strategies that can yield success with different/same probability (ie underhand vs overhand shots). It is possible these are used in different trials by the same subject. Unfortunately, the only data available is number of successes per session (20 trials).

I do not think I am looking for "ways to construct probability models of underdispersed phenomena", at least not in general. I am not interested in only describing the data, rather for a process that can result in this type of data. The goal is to elucidate what may actually be being measured here if not max/asymptotic/plateau performance level.

To clarify what I mean by "process", I am thinking that a monte carlo simulation can be created using some combination of if/then statements and (possibly multiple per trial) samples of correct/incorrect actions, states, and/or events that occur with various probabilities. However, there may be other ways of modeling this.

Edit3: @gung I do not think we will be able to identify a process/mechanism from this data alone, but we can hypothesize a few consistent with the data. These will then make predictions regarding other/more detailed measurements (eg trial-to-trial scores) before running the study. This is useful because it suggests what it is important to look for and record when performing the experiments.

I thought of another possible mechanism. The model below simulates a situation where the subject is "satisfied" after a threshold # of successes (here thresh=12). The output shown had variance=0.495. If this model were accurate, rather than performance, these experiments appear to measure some kind of motivation threshold. This would be completely different than measuring a skill level, and really alter how these results are interpreted. However, this model predicts many more successes at the beginning of the session than the end. While I do not have actual data recorded regarding this, the prediction is inconsistent with my memory/impression of what unfolded. If anything, I suspect the opposite would be true.

I am looking for further ideas on what the explanation may be as I could not find any hints in the literature.

p.motivated=.9; p.unmotivated=.1; n=20; thresh=12; sessions=15
results<-matrix(nrow=sessions)
for(s in 1:sessions){
  session.dat<-matrix(nrow=n,0)
  for(t in 1:n){
    if(sum(session.dat)<thresh){
      session.dat[t]<-sample(c(0,1),1,prob=c(1-p.motivated,p.motivated))
    }else{
      session.dat[t]<-sample(c(0,1),1,prob=c(1-p.unmotivated,p.unmotivated))
    }
  }
  results[s]<-sum(session.dat)
}

hist(results,breaks=seq(0,20,by=1))
var(results)

enter image description here

$\endgroup$
  • $\begingroup$ The only question that appears in all this is #1, whose answer you obviously already know. What do you really want to ask? Whether this dataset looks significantly underdispersed (as suggested by the title)? What does the title mean by "mechanisms"? $\endgroup$ – whuber Nov 7 '14 at 18:58
  • $\begingroup$ Thanks for the update. "Max performance level achieved" may be a key phrase, because it suggests that data might not appropriately modeled as an iid Binomial sample. Unfortunately, your question about "mechanisms" cannot be answered until you disclose what the data really represent! The factors that might affect consistency of free throw performance will be largely different than those that affect consistency of ranks in a series of races, for instance. Having said that, there are plenty of ways to construct probability models of underdispersed phenomena. Maybe that's what you are looking for? $\endgroup$ – whuber Nov 7 '14 at 19:33
  • $\begingroup$ We may not be able to identify the mechanism that produces your results. The autocorrelation idea is interesting. To investigate it you will need to get the trial by trial data from your study, or run a new study to get such data. $\endgroup$ – gung - Reinstate Monica Nov 7 '14 at 20:18
  • $\begingroup$ @whuber On rereading I realized I do not follow you here. Can you expand on this statement "Max performance level achieved"... suggests that data might not appropriately modeled as an iid Binomial sample. $\endgroup$ – Livid Nov 8 '14 at 0:26
  • $\begingroup$ If the data reflect maxima of sets of values, then they will tend to be narrowly dispersed and a little negatively skewed. A binomial model will not describe them. $\endgroup$ – whuber Nov 8 '14 at 2:05
3
+100
$\begingroup$

This kind of underdispersion can happen if the subject needs to calibrate themselves to the task on any given day. This implies that the probability of success is not constant but changes according to a fixed pattern. The following sequence of success probabilities (one for each of the 20 trials, to be used every day) produces a distribution like the one shown. This is known as a Poisson-Binomial distribution.

0.00,0.00,0.00,0.00,0.03,0.10,0.34,0.72,0.97,0.99,0.99,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00,1.00

Besides this particular sequence, any sequence of probabilities whose mean is 0.65 and whose variance is 0.2 will work.

$\endgroup$
  • 1
    $\begingroup$ Thanks, this sounds plausible. It is as if there is a warm up period. To model this each subject would have p.initial, p.final, and some intra-session "learning" constant. On first glance, it seems the parameters of such a model will be underdetermined by the data available (successes/session). I do not follow where you are getting variance = 0.2 though. $\endgroup$ – Livid Nov 10 '14 at 20:08
  • 1
    $\begingroup$ Variance = 0.2 comes from matching the variance of the Poisson-Binomial to the empirical variance of the data provided. $\endgroup$ – Tom Minka Nov 10 '14 at 20:42
  • $\begingroup$ I've realized that if we drop the dependance on # successes my "motivation threshold" model is the simplest example of a poisson-binomial (simply two different probs in some combination). If order is ignored, such a model can be interpreted in multiple ways such as warm up, switching strategies, etc. This may be the best approach given the lack of constraints on the trial-to-trial sequence. Thank you for the name of this. $\endgroup$ – Livid Nov 11 '14 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.