I have a sample, set of outcomes of some random variable. I divide it into "clusters", using some determinate approach. One of the clusters considered to be "correct", usually it is the one that has the most number of outcomes in it, but not always. I want to mark each cluster with some "confidence level", based on the number of outcomes it has. The formulae should depend on the number of outcomes in each cluster totally, and in the current one.

Say, if the outcome numbers, for each cluster sequentially, are: 220, 31, 28, 21, then the first cluster should have very high confidence level, something close to 100%, but not exactly 100%. But if the distribution is 113, 110, 106, 103, then all the confidences should be almost equal - and not even close to 100% - since there are several similar clusters, we cannot mark each of them as "probably correct".

I cannot use just relative proportion to the maximum, because it will cause highest cluster to have always exactly 100% confidence level, and it shouldn't be.

Is there some approach, or formulae, that gives such estimations? Thank you.


migrated from math.stackexchange.com Aug 2 '11 at 21:02

This question came from our site for people studying math at any level and professionals in related fields.

  • $\begingroup$ The most direct measure would be the proportion of outcomes in that cluster, not relative to the maximum but to the sum, so in your examples it would be 0.733, 0.103, 0.093, 0.070 and 0.262, 0.255, 0.245, 0.238, respectively. If that doesn't suit your purposes, you should say in what way you want it to differ from that. Would you want the cluster with 220 to have a higher confidence level than 73%? $\endgroup$ – joriki Aug 2 '11 at 20:47
  • $\begingroup$ @joriki This approach is sensitive to the sum, not to the actual numbers and their differences. Firstly, for first example 73% is too low, it should be more than 90%. But this sentence is not so mathematically strict. Secondly, what's more important, if there's one cluster with 100 outcomes, and ten with 10 outcomes each, then first will be measured as 50% confidence, when in reality in such situation first cluster is almost always correct. So, more "noise" - more confidence that not-noisy cluster is correct. $\endgroup$ – Skiminok Aug 2 '11 at 20:53

You could use the conditional probabilities for the outcomes to be the correct outcomes under a prior assumption that the correct outcome occurs with some probability $p$ and all incorrect outcomes occur with probability $q=(1-p)/(n-1)$, where $n$ is the number of different outcomes. Then the probability for outcomes $a_i$ with sum $\sum_ia_i=S$ if answer $k$ is correct is proportional to $p^{a_k}q^{S-a_k}$, so you could assign confidence levels

$$c_k=\frac{p^{a_k}q^{S-a_k}}{\sum_i p^{a_i}q^{S-a_i}}=\frac{p^{a_k}q^{-a_k}}{\sum_i p^{a_i}q^{-a_i}}\;.$$

You can choose the parameter $p$ according to your needs. For $p=q=1/n$, you'll get $c_k=1/n$ (which makes sense, since if you assume that people are just guessing, no amount of clustering will raise your confidence in one of the outcomes). For $p$ near $1$, you'll get sharply peaked confidence levels even for moderate differences in the outcome counts. In the limit $p\to1$, $q\to0$ the confidence level for the outcome with the highest count will go to $1$ and the others will go to $0$, since you can multiply through by the lowest number of factors of $q$ and the other terms in the sum go to zero.

If you want the confidence level in your first example to be noticeably different from $100\%$, you'd have to choose $p$ quite close to $1/n=25\%$. Here are some values for your examples:

$$ \begin{array}{|c|c|c|c|c|c|} p&q&220&31&28&21\\ \hline 0.253&0.249& 0.879&0.043&0.041&0.037\\ \hline 0.256&0.248&0.994&0.002&0.002&0.002\\ \hline 0.259&0.247&1.000&0.000&0.000&0.000 \end{array} $$

$$ \begin{array}{|c|c|c|c|c|c|} p&q&113&110&106&103\\ \hline 0.253&0.249& 0.270&0.258&0.242&0.230\\ \hline 0.256&0.248&0.291&0.264&0.233&0.212\\ \hline 0.259&0.247&0.312&0.270&0.224&0.194\\ \hline 0.265&0.245& 0.354&0.280&0.204&0.162\\ \hline 0.280&0.240&0.458&0.288&0.156&0.098\\ \hline 0.310&0.230&0.632&0.258&0.078&0.032 \end{array} $$

  • $\begingroup$ Why do you have the same photo as a different user on math.SE? $\endgroup$ – cardinal Aug 2 '11 at 22:27
  • $\begingroup$ My question is: why do I have a different name here? :-) I'm the same person, but here my real name is being displayed and on math.SE my screen name is displayed -- I have no idea why this happened; if you know how I can change it I'd appreciate it :-) $\endgroup$ – joriki Aug 2 '11 at 22:29
  • $\begingroup$ Interesting -- the screen name was set to my real name. That shouldn't happen. $\endgroup$ – joriki Aug 2 '11 at 22:31
  • $\begingroup$ Well, that's a good alternate question! No, I don't know the answer. But, perhaps the moderators do. It's probably some account merging issue. I'll flag them and maybe they can help. $\endgroup$ – cardinal Aug 2 '11 at 22:31
  • 1
    $\begingroup$ @joriki, accounts issue aside, welcome to stats.SE! $\endgroup$ – mpiktas Aug 3 '11 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.