Measuring how well a new vote fits to a model of existing votes

Question

My knowledge in statistics is very limited, so I hope this is actually an easy question. I am working on a kind of survey application where users either vote between a finite number of discrete choices, or on continuous values in a certain range. My goal is to assess each vote as they come in on how well they fit with all received votes so far. More precisely, I need a way to determine how likely it is that the incoming vote is generated by the underlying model which I have to infer from all existing votes.

For the discrete case, my idea was to simply infer the discrete distribution directly from how often each choice has been voted so far, then using this distribution to get the probability of the new vote. The problem is that e.g. for the second vote I would get a probability of zero if it was different from the first vote. The probability should only approach zero if there were an increasing number of votes already and nobody would have voted for this choice. I am not sure how to model it so it behaves like this.

For the continuous case, it is fair to assume that the votes are distributed normally. My idea here was to infer the model from mean and variance of all existing votes and using it to get the likelihood of the new vote. But I just came along this and was wondering if I would need to incorporate that somehow?

Hope you can point me in the right direction here.

Answer 1

Figuring out how likely a given value is to be produced by a given distribution (e.g., given that a mystery value was drawn from a standard normal distribution, how likely is to be positive?) is easy enough. But the opposite problem, figuring out how likely a given distribution is to be responsible for a given value (e.g., given that I got the value 3, how likely is the mystery distribution it came from to be a standard normal distribution?) is quite a bit more complex. You'd need to begin by specifying a universe of models (or just distributions) to consider, which could be a small finite set or an infinite set.

Perhaps there's a more direct way to do what you want to do here. What problem are you trying to solve?

Edit: After further clarification in the comments, here's a more specific suggestion. Use a Bayesian method in which you set hyperparameters to encode your starting beliefs about the voting distribution, measure each vote's distance from the distribution so far with the likelihood, and use each vote you get to update the distribution. Specifically, for the continuous case, suppose votes come from a normal distribution with mean μ and standard deviation σ, μ is drawn from a fairly wide and flat normal distribution, and σ is drawn from a wide uniform distribution from .0001 to 100 or so. (You can also try the conjugate prior for the normal distribution.) For the discrete case, suppose votes come from a categorical (multinomial) distribution whose parameters come from a Dirichlet distribution.

The benefit of using these Bayesian methods over the methods you described is that they do something reasonable even when you don't have any data yet or only a little data (instead of, say, assuming that categories yet unseen have probability 0) and they provide a natural transition (as the sample size increases) into using the data alone to estimate the distributions.

Answer 2

For the discrete case, my idea was to simply infer the discrete distribution directly from how often each choice has been voted so far, then using this distribution to get the probability of the new vote.

How about use:

$$\frac{\text{times choice has been made}+1}{\text{total choices that have been made} + \text{total number of categories}}$$

This seems to have the properties you're looking for (doesn't approach zero until you have significantly more observations than categories).

I think this actually corresponds to Bayesian predictive probabilities with a multinomial model, and a Dirichlet prior.

The analogous thing for the continuous case would be Bayesian inference with a normal model, and conjugate priors (normal for the mean, inverse-gamma for the variance). This does indeed give you a Student's $t$ distribution for your predictive distribution. I recommend Peter Hoff "A First Course in Bayesian Statistical Methods" for more info.