Suppose I have realized n predictions $p_1, \dots, p_n$ using a classification method (for instance a logistic regression or a linear discriminant analysis). Each prediction is associated with a standard error $\sigma_1, \dots, \sigma_n$.

I want to estimate the mean prediction and a confidence interval for it at a certain level. I'm wondering what the best approach is to tackle this problem. Should I compute a classic sample mean and a sample standard deviation from the predictions? Or would it be more accurate to compute a kind of pooled estimate using the individual standard errors?

Another possibility to get the bounds of the confidence interval would be to compute the empirical quantiles but I'm not sure it would be statistically right. What do you think?

That's right, my n predictions refer to the same target which is a binary variable. But I may have forgotten to precise that it was the same model (in this case a fitted logistic regression) that produced all the predictions, that's why I wasn't sure whether a pooling method was appropriate.

Thanks for the information. I didn't think of considering these predictions realizations of a mixture model. I rather looked into pooling methods and the fact is I found many different formulas to combine the predictions into a "consensus" so I was a bit confused.

I also had difficulty with building the confidence interval for $p_c$ since I didn't know the quantiles of the distribution of the predictions. Of course, I thought of using the quantiles of the standard normal distribution, but in some cases, the predictions reduced to a pointwise estimation (for a specific sub-sample of the population, all individuals were similar and the prediction was the same for all of them), so this approximation should be very bad. Can the normal quantiles be used in this very situation?

Another thing bugged me: on some sub-samples, the discretized predictions returned only 0's or only 1's so I changed them back to probabilities in order to get lightly different values and to be able to estimate a mean prediction. But I'm not sure this was the right approach to follow...


I assume the $n$ predictions refer to the same target, a binare variable $Y \in \{0,1\}$, and your goal is to merge it into a single prediction. Note that the $p_i$s are (Bernoulli) distributions. The stats literature knows many ways of aggregating a set of $n$ distributions into a single ("conensus") one, see e.g. http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.ss/1177013825

Taking the mean prediction is called a linear prediction pool (LOP). The mean and variance of the LOP are as follows (http://en.wikipedia.org/wiki/Mixture_distribution):

Mean = $p_c = \sum_{i=1}^n \omega_i p_i,$

Variance = $\sigma^2_c = \sum_{i=1}^n \omega_i \sigma^2_i + \sum_{i=1}^n \omega_i (p_i-p_c)^2,$

where the $\omega_i$s are the weights for each model (required to be nonnegative and sum to one), and $\sigma_i^2 = p_i(1-p_i)$ is the variance of prediction $i$.

Intuitively, the first sumand for the variance term reflects the average variance, whilst the second term is the disagreement among the $n$ individual forecasts. Both contribute to the uncertainty of the combined forecast.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.