Combining probabilities for class prediction Before I release a new web service, I am trying to develop my data analytics skills so I can monitor the performance of the service. I may end up using and external service (eg: Google Analytics) or an analytics library, but I need to understand how the algorithms work before I can be confident I am using and interpreting the analytics correctly. The best way I learn is to build software, but I am stuck on a (probably simple to experienced analysts) problem:
I have a dataset of 10,000 items.
There are 100 labelled with class A (the items can have multiple classes).
I note that 60 of the class A items have attribute X. I also count the other significant attributes that seem to correlate with items of class A.
I might then think that any new item with attribute X has a 60% chance of being a class A item (ignoring the influence of other attributes for the present).
But I also calculate that 20% of the items of the whole dataset have attribute X.
How do I calculate the change in my prediction probability? I suspect a Bayesian approach would be suitable but the results do not look right.
If I make the same calculations with other classes and other attributes, how do I merge the probabilities? (eg: to find the probability of a new item with attributes X,Y having class A, I look up P(A|X)=0.3 and P(A|Y)=0.5, then...). I tried the p=ab/(ab+(1-a)(1-b)) merging formula but this seems wrong.
Some gentle guidance would be appreciated.
 A: If a new observed item s has the attribute X, then the probability that item s belongs to class A according to the Bayes' theorem is equal to:
\begin{equation}
p(s \in A | X, I) = p(s \in A | I) \times \frac{ p(X | s \in A, I) }
{p(X | I) }
\end{equation}
where the $p(s \in A | I)$ is called the prior probability that before seeing any data, in this case before knowing that $s$ possess attribute $X$, how much we'd bet that $s$ belongs to $A$. Well it all depends on your knowledge or previous information about this system, which is encoded in $I$. But the second term is called the likelihood of your data which says given that $s$ belongs to $A$, then how probable it's that $s$ has attribute $X$. In your case it's simply a Bernoulli trial which the chance of success is $p$. Here we have $p \approx 0.6$. I'd write it however like this:
\begin{equation}
p(X | s\in A, I) = \frac{m + 1}{n + 2}
\end{equation}
which here $m=60$ and $n$ is the size of your dataset and is 100. But why I add some extra values to the denomerator and numerator? it's simply called Laplace smoothing and you can learn about it here. 
For the prior probability $p(s\in A| I)$, if you're completely ignorant about it, you can assign $1/m$ which $m$ is the number of different classes. Or $p(s \in A| I) = 1/2$ if the classes are not mutually exclusive. However, if you gained new information about the chance that an item is in class $A$, then we can decide on other functional forms for $p(s \in A|I)$. 
The term $p(X|I)$ in the denumerator of the Bayes' equation is what is called in Bayesian jargon "evidence". Here you mentioned that 20% of all items in your dataset have attribute $X$. So this is where this information should be incorporated. $p(X|I)$ tells what's the probability that item $s$ has feature $X$ no matter what class it belongs to. More formally, 
\begin{equation}
p(X|I) = \sum_C p(X|s \in C,I)p(s \in C, I)
\end{equation}
For calculating the evidence you must use your information about any other classes that $s$ may belong to. But you already have calculated that 20% of 10,000 items have attribute $X$. We know that 60 of these 2,000 items with attribute X are from class A. If I just say that the remaining 1940 items are not labled $A$, then: 
$p(X | I) = 0.5 \times \frac{61}{102} + 0.5 \times \frac{1941}{994 2}$
Now using all these numbers, we can work out the posterior probability that item $s$ belongs to class $A$:
\begin{equation}
p(s \in A | X, I) = 0.5 \times \frac{62/102}{(0.5 \times \frac{61}{102} + 0.5 \times \frac{1941}{9942})} = 0.754
\end{equation}
Note that if your observation was that 80 out of 100 of the items of $A$ have $X$, then $p(s \in A | X, I) = 0.804$. Even if 100 out of 100 were labled of having $X$, $p(s \in A | X, I) = 0.838$ just because your sample size compare to the rest of the dataset is relatively small. I'd however suggest to calculate $p(X|I)$ use all the information you have for the rest of the classes, then your calculation would be more acceptable. 
I hope it was helpful :D
