# Aggregation of propensity scores with varying reliability

When trying to estimate the number of sampling units with an attribute, is there a good algebraic way to aggregate over propensity scores for that attribute which each have their own error? For example, when the propensity scores may be calculated with varying amounts of information from each sampling unit and each has its own standard error on that propensity.

Or they could be expected values of beta distributions.

In the latter case I know I can simulate beta-bernoulli outcomes from each sampling unit and add up the results many times; but is there a consistent estimator of the result of this difficult to scale process?

In short, how do people aggregate propensity scores of varying reliabilities?

Edit:

I suppose I worded it poorly; the data I have is all either binary or categorical and each observation is accompanied by the probability it was observed correctly. So suppose I have 5 persons; 3 of which had a value of 1 for an attribute, 2 of which had a value of 0 for that attribute, each of which had a probability .8,.81,.82.,.83,.84 respectively of being observed correctly. What is the expected value of p(having that attribute)?

-
I don't understand what you're trying to do here. To estimate propensity scores you need to know whether each individual received the treatement or not, in which case you can simply count up how many received the treatment. Why do you want to aggregate propensity scores? –  onestop Nov 14 '10 at 14:04
I suppose I worded it poorly; the data I have is all either binary or multinomial and each observation is accompanied by the probability it was observed correctly. So suppose I have 5 persons; 3 of which had a value of 1 for an attribute, 2 of which had a value of 0 for that attribute, each of which had a probability .8,.81,.82.,.83,.84 respectively of being observed correctly. What is the p(having that attribute)? –  Patrick McCann Nov 16 '10 at 17:27

I'm not sure I understand your question, but you have no other answers, so I thought I'd give it a shot and revive this topic: If I understand you correctly for a number of different sampling units (say people) you have estimates that they have given attribute, e.g. there is a probability that they have eaten turkey today - Pr(Turkey). You also have an estimate of the error in your record of Pr(Turkey). Thus, you expect that Joe has a Pr(Turkey) of 50% +/- 10%. Given information of this sort, I think you are interested in the expected number of your sampling units that will have eaten Turkey.

I believe you can simply sum the propensity scores and you'll get the expected number of sampling units with the attribute so long as the error of the estimate is symmetrical. In short, I'm proposing that the errors of the estimate become unimportant when you aggregate. The only time the variance is going to matter is if you want the expected variance of your aggregate score.

Edit: Given the edit to the question I think the approach outlined above may still be salvageable, especially for binary observations. Taking the supposition that we have 5 persons; 3 of which had a value of 1 for an attribute, 2 of which had a value of 0 for that attribute, each of which had a probability .8,.81,.82.,.83,.84 respectively of being observed correctly. We want to find the expected value of p(having that attribute). We can invert the probability of having that attribute for those who were judged not to have the attribute such that we can phrase each of the 5 observations in terms of their probability of having the attribute, .8, .81, .82, (1 - .83) .17, and (1 - .84) .16, respectively. Thus the average Pr of having the attribute should be (.80+.81+.82+.17+.16)/5 = .552. Two problems exist with this approach.

1. It assumes that probabilities are equally probable (if you will). That is, that that a certainty of .95 and .75 means that the average probability of having the attribute is .85. It may be the case that averaging on a linear scale isn't the right sort of thing to do. Perhaps these probabilities should be converted to logits then back, in which case the average probability of .95 (logit: 2.94) and .75 (logit: 1.10) is .88 (logit: 2.02).
2. It has little to say about the case of non-binary classifications. Should one judge that if the categorization is .90 likely to be correct and there are two other options that there is a .05 chance it is one and .05 chance it is the other? Should base rates for the other options be considered? Etc.
-
I have realizations of Turkey and probabilities I observed that correctly; I edited the question to better reflect this. Thanks! –  Patrick McCann Nov 16 '10 at 17:28
Eventually figured out I should do an inverse Markov transition step. –  Patrick McCann Mar 15 '11 at 16:57