Let's say we have a mathematical model that provides the probability of finding oil at a location in terms of a system of 10 bins with probabilities going from very low, say 2%, to 20% for the best bin. Let's also assume that one of the measures we use to monitor the performance of this model is the Powerstat (Gini coefficient). [To be sure: This question is not about the Powerstat's suitability for this task]. Presumably, it would provide an indication how well the model is able to differentiate future successful oil strikes from failures.
Let's say we have had a drilling campaign with 1000 trials. Before drilling, those 1000 locations were divided by the model over the 10 bins, for instance like {50,75,100,125,150,150,125,100,75,50}. Let's say we have found 100 oil wells among those 1000 trials distributed like {0, 2, 6, 14, 13, 18, 11, 13, 10, 8}. We can now calculate the Powerstat given this result.
We want to determine whether or not the Powerstat of the model for this given data set of1000 drillings significantly deviates from what we would expect to find if the model works as described above. We could exactly calculate (at least in theory) the Powerstat for all possible outcomes and their probabilities or estimate that by means of Monte Carlo simulation and determine a confidence interval in this way.
The main discussion we have is whether in this process we should use the information that we have in fact found 100 oil wells or whether we should not. So, in the MC case, should we generate random drilling outcomes based on a binomial model, the bin probabilities and the bin populations and throw away all results not totaling to 100 or should we use all samples regardless of their total number of oil strikes?
In one view, the 100 oil well number is a more or less arbitrary outcome of the campaign and is just one realization of a large range of conceivable numbers (in fact anything between 0 and 1000, with varying probabilities). Using the given total of 100 would be just as arbitrary using a bit of information as would be, for instance, using information on bin 10 (containing 8 oil strikes), or the totals of bin 1-5 and 6-10 (each of which would lead to different intervals).
The other view is that we should use all information we have; that having an outcome of 100 says something already about the model quality and that we should take that into account.
So, the question is: which view is correct?
PS: I had difficulties coming up with a descriptive title. Any improvement is welcome.
