Question in brief: I am attempting to determine whether statistical significance can be measured merely from counts generated from iterating the same procedure, but where the probability of success is unknown both longitudinally and cross-sectionally. If so, is there an analytic solution? If not, can you provide guidance on a numerical solution?
Question in full: The following experiment has been conducted. There are 33 children on a sports team, and once a day a coach is tasked with placing each child into one of two groups: group one, which is the group that will actually play the match, and group two, which consists of those children who will be sent home. The coach's objective is to win the match, and he can do that by selecting as many (or as few) children as he desires (i.e., $[0,33]$, where a selection of 0 would indicate a forfeit, though empirically this never occurred; also, the game doesn't require a minimum number of participants). This task is performed each day of the school year (175 iterations). We can assume (for reasons unnecessary to expound upon here) that the coach does not learn; i.e., each time he performs this task it is independent of past attempts. Further, the same 33 children are always decided upon and the children's perceived value remains the same each day (i.e., the children do not improve, etc.).
Thus there are two things we have measured: (1) how many children the coach selects in each iteration (i.e., the sum of each "Iteration n" column below), and (2) the number of times each child is selected (i.e., the "Sum of count" column below). The table below shows a sample of the collected data:
Child ID | Iteration 1 | Iteration 2 | ... | Sum of count ---------------------------------------------------------- 01 | 0 | 0 | ... | 0 02 | 0 | 1 | ... | 28 03 | 1 | 1 | ... | 175 04 | 0 | 0 | ... | 0 . | . | . | ... | . . | . | . | ... | . . | . | . | ... | . 33 | 1 | 1 | ... | 172
where if the child was selected in that iteration he/she received a 1, otherwise a 0. I want to know whether, for example, child 02 was selected a statistically-significant number of times (i.e., is 28 significant?).
Thoughts: Because the selection of a child into group one is a binary event ("yes" or "no"), I thought I could compare each child's count against the binomial CDF. But this doesn't work because I don't know ex-ante the probability of a child being selected; empirically we see that somewhere around only 3 to 6 children are generally selected (meaning around 27 to 30 are placed into group two in each iteration). I think this is effectively because each child is not independent of the others: some are "fast runners," some are "the big guy," etc., and thus the coach is selecting the best from a smaller group of categories. But this is not required in the setup, and I don't know which category the coach will care about on a particular iteration. I thought a Poisson distribution might be appropriate, but I don't know ex-ante lambda (the average number of children selected in an iteration). There are some similarities to the Mann-Whitney U test, but in the present setup there is no rank: group one or group two is all I have. And so on.
Thanks in advance,