Is it possible to empirically discriminate distributions of the probability of a binary outcome? I am interested in systems with an observable binary outcome (e.g. Success, Failure). The individual cases are usually modelled (e.g. using logistic regression) as having an estimated probability of success. 
This probability of success could be viewed as just a side effect of observational ignorance:


*

*Each case is predetermined as a Success or Failure. We just don't
know until we look, and the set of covariates doesn't allow
partitioning the cases into exclusively S or F cases, so every group
of cases we can identify by the covariates has a mixture of S and F.


Or the probability of success could be viewed as a real but unobservable property of each case:


*

*Cases are not predetermined to be S or F, rather that status is
determined by some outcome process such that when we get around to
observing it, each case will be S or F. Each case has an unobservable
probability of Success that determines the probability that the
observed outcome is S and we have no access to any better
information.


If we take the latter interpretation, it makes sense to talk about the distribution of the probability of S over cases. Given some group of cases with a known group probability of S it seems to me that there are an infinite number of distributions of case-level P(S) that would yield the same group-level P(S), i.e. all distributions with the same expectation of P(S). This would range from all cases having identically the same P(S) through to each case having P(S) either 1 or 0, with the group-level P(S) determined by the fraction of cases having case-level P(S) = 1.
So the question is: Assuming each case has it's own fixed probability of Success and only the outcomes can be observed, is it possible to empirically discriminate between different distributions of probability of Success over cases? I strongly suspect the answer is 'No'.
If the distributions can't be empirically discriminated, is it possible to relax some aspect of the above assumptions (e.g. by allowing partial observability of the process that translates the unobservable probability of Success into the observable outcome) so that the distributions of probability of Success are empirically discriminable? That is: What would you have to do to make it possible to empirically discriminate between different distributions of probability of Success over cases?
Thanks
Ross
 A: "Assuming each case has it's own fixed probability of Success and only the outcomes can be observed, is it possible to empirically discriminate between different distributions of probability of Success over cases?" 
If each case has it's own fixed probability of success then that probability is indistinguishable from the outcome. That is $P(X_i = 1)$ must be either 1 or 0 because there is no chance for it to be otherwise - because by your definition each $S_i$ is unique and paired directly with an $X_i$. So, yes it is possible to model some distribution over $\bf{S}$, but all you've done is add another layer of meaningless abstraction. Instead of your data being Bernoulli distributed, your parameters are Bernoulli distributed, and we can described the parameters of some hyper-distribution with co-variates, probably with logistic regression. At a certain point, you need some concept of replication to be able to resolve a statistic. And here all a statistic means is some shorthand summary of data or in your abstract case of parameters. Otherwise, all you have is one long series of unique events with no more sufficient or compact way to describe them then to simply observe them.
A: I think that what you would have to do in order to empirically discriminate between the $P(S_{i})$ is find some characteristic which varies among the i's that correlates with higher or lower $P(S_i)$.
For example, take wine making.  It takes a long time to make wine, and it is not immediately obvious through any known chemical testing to determine if a particular year/vintage will be "good."  However, in the book "Supercrunchers" a formula is described (see review at http://www.fool.com/investing/small-cap/2007/10/31/foolish-book-review-super-crunchers.aspx), which converts temperature and rainfall into a quality rating.  Of course there are many other hidden variables, and it does not guarantee an exact level of quality.  However, I hope that this example shows how using salient features of a process can help one forecast the end result of that process.
