Suppose I have realized n predictions $p_1, \dots, p_n$ using a classification method (for instance a logistic regression or a linear discriminant analysis). Each prediction is associated with a standard error $\sigma_1, \dots, \sigma_n$.
I want to estimate the mean prediction and a confidence interval for it at a certain level. I'm wondering what the best approach is to tackle this problem. Should I compute a classic sample mean and a sample standard deviation from the predictions? Or would it be more accurate to compute a kind of pooled estimate using the individual standard errors?
Another possibility to get the bounds of the confidence interval would be to compute the empirical quantiles but I'm not sure it would be statistically right. What do you think?
That's right, my n predictions refer to the same target which is a binary variable. But I may have forgotten to precise that it was the same model (in this case a fitted logistic regression) that produced all the predictions, that's why I wasn't sure whether a pooling method was appropriate.
Thanks for the information. I didn't think of considering these predictions realizations of a mixture model. I rather looked into pooling methods and the fact is I found many different formulas to combine the predictions into a "consensus" so I was a bit confused.
I also had difficulty with building the confidence interval for $p_c$ since I didn't know the quantiles of the distribution of the predictions. Of course, I thought of using the quantiles of the standard normal distribution, but in some cases, the predictions reduced to a pointwise estimation (for a specific sub-sample of the population, all individuals were similar and the prediction was the same for all of them), so this approximation should be very bad. Can the normal quantiles be used in this very situation?
Another thing bugged me: on some sub-samples, the discretized predictions returned only 0's or only 1's so I changed them back to probabilities in order to get lightly different values and to be able to estimate a mean prediction. But I'm not sure this was the right approach to follow...