Aggregation of propensity scores with varying reliability

Question

When trying to estimate the number of sampling units with an attribute, is there a good algebraic way to aggregate over propensity scores for that attribute which each have their own error? For example, when the propensity scores may be calculated with varying amounts of information from each sampling unit and each has its own standard error on that propensity.

Or they could be expected values of beta distributions.

In the latter case I know I can simulate beta-bernoulli outcomes from each sampling unit and add up the results many times; but is there a consistent estimator of the result of this difficult to scale process?

In short, how do people aggregate propensity scores of varying reliabilities?

Edit:

I suppose I worded it poorly; the data I have is all either binary or categorical and each observation is accompanied by the probability it was observed correctly. So suppose I have 5 persons; 3 of which had a value of 1 for an attribute, 2 of which had a value of 0 for that attribute, each of which had a probability .8,.81,.82.,.83,.84 respectively of being observed correctly. What is the expected value of p(having that attribute)?

Answer 1

I'm not sure I understand your question, but you have no other answers, so I thought I'd give it a shot and revive this topic: If I understand you correctly for a number of different sampling units (say people) you have estimates that they have given attribute, e.g. there is a probability that they have eaten turkey today - Pr(Turkey). You also have an estimate of the error in your record of Pr(Turkey). Thus, you expect that Joe has a Pr(Turkey) of 50% +/- 10%. Given information of this sort, I think you are interested in the expected number of your sampling units that will have eaten Turkey.

I believe you can simply sum the propensity scores and you'll get the expected number of sampling units with the attribute so long as the error of the estimate is symmetrical. In short, I'm proposing that the errors of the estimate become unimportant when you aggregate. The only time the variance is going to matter is if you want the expected variance of your aggregate score.

Edit: Given the edit to the question I think the approach outlined above may still be salvageable, especially for binary observations. Taking the supposition that we have 5 persons; 3 of which had a value of 1 for an attribute, 2 of which had a value of 0 for that attribute, each of which had a probability .8,.81,.82.,.83,.84 respectively of being observed correctly. We want to find the expected value of p(having that attribute). We can invert the probability of having that attribute for those who were judged not to have the attribute such that we can phrase each of the 5 observations in terms of their probability of having the attribute, .8, .81, .82, (1 - .83) .17, and (1 - .84) .16, respectively. Thus the average Pr of having the attribute should be (.80+.81+.82+.17+.16)/5 = .552. Two problems exist with this approach.

1. It assumes that probabilities are equally probable (if you will). That is, that that a certainty of .95 and .75 means that the average probability of having the attribute is .85. It may be the case that averaging on a linear scale isn't the right sort of thing to do. Perhaps these probabilities should be converted to logits then back, in which case the average probability of .95 (logit: 2.94) and .75 (logit: 1.10) is .88 (logit: 2.02).
2. It has little to say about the case of non-binary classifications. Should one judge that if the categorization is .90 likely to be correct and there are two other options that there is a .05 chance it is one and .05 chance it is the other? Should base rates for the other options be considered? Etc.