3
$\begingroup$

Before I release a new web service, I am trying to develop my data analytics skills so I can monitor the performance of the service. I may end up using and external service (eg: Google Analytics) or an analytics library, but I need to understand how the algorithms work before I can be confident I am using and interpreting the analytics correctly. The best way I learn is to build software, but I am stuck on a (probably simple to experienced analysts) problem:

I have a dataset of 10,000 items. There are 100 labelled with class A (the items can have multiple classes). I note that 60 of the class A items have attribute X. I also count the other significant attributes that seem to correlate with items of class A.

I might then think that any new item with attribute X has a 60% chance of being a class A item (ignoring the influence of other attributes for the present).

But I also calculate that 20% of the items of the whole dataset have attribute X. How do I calculate the change in my prediction probability? I suspect a Bayesian approach would be suitable but the results do not look right.

If I make the same calculations with other classes and other attributes, how do I merge the probabilities? (eg: to find the probability of a new item with attributes X,Y having class A, I look up P(A|X)=0.3 and P(A|Y)=0.5, then...). I tried the p=ab/(ab+(1-a)(1-b)) merging formula but this seems wrong.

Some gentle guidance would be appreciated.

$\endgroup$

1 Answer 1

3
$\begingroup$

If a new observed item s has the attribute X, then the probability that item s belongs to class A according to the Bayes' theorem is equal to:

\begin{equation} p(s \in A | X, I) = p(s \in A | I) \times \frac{ p(X | s \in A, I) } {p(X | I) } \end{equation}

where the $p(s \in A | I)$ is called the prior probability that before seeing any data, in this case before knowing that $s$ possess attribute $X$, how much we'd bet that $s$ belongs to $A$. Well it all depends on your knowledge or previous information about this system, which is encoded in $I$. But the second term is called the likelihood of your data which says given that $s$ belongs to $A$, then how probable it's that $s$ has attribute $X$. In your case it's simply a Bernoulli trial which the chance of success is $p$. Here we have $p \approx 0.6$. I'd write it however like this:

\begin{equation} p(X | s\in A, I) = \frac{m + 1}{n + 2} \end{equation}

which here $m=60$ and $n$ is the size of your dataset and is 100. But why I add some extra values to the denomerator and numerator? it's simply called Laplace smoothing and you can learn about it here.

For the prior probability $p(s\in A| I)$, if you're completely ignorant about it, you can assign $1/m$ which $m$ is the number of different classes. Or $p(s \in A| I) = 1/2$ if the classes are not mutually exclusive. However, if you gained new information about the chance that an item is in class $A$, then we can decide on other functional forms for $p(s \in A|I)$.

The term $p(X|I)$ in the denumerator of the Bayes' equation is what is called in Bayesian jargon "evidence". Here you mentioned that 20% of all items in your dataset have attribute $X$. So this is where this information should be incorporated. $p(X|I)$ tells what's the probability that item $s$ has feature $X$ no matter what class it belongs to. More formally,

\begin{equation} p(X|I) = \sum_C p(X|s \in C,I)p(s \in C, I) \end{equation}

For calculating the evidence you must use your information about any other classes that $s$ may belong to. But you already have calculated that 20% of 10,000 items have attribute $X$. We know that 60 of these 2,000 items with attribute X are from class A. If I just say that the remaining 1940 items are not labled $A$, then:

$p(X | I) = 0.5 \times \frac{61}{102} + 0.5 \times \frac{1941}{994 2}$

Now using all these numbers, we can work out the posterior probability that item $s$ belongs to class $A$:

\begin{equation} p(s \in A | X, I) = 0.5 \times \frac{62/102}{(0.5 \times \frac{61}{102} + 0.5 \times \frac{1941}{9942})} = 0.754 \end{equation}

Note that if your observation was that 80 out of 100 of the items of $A$ have $X$, then $p(s \in A | X, I) = 0.804$. Even if 100 out of 100 were labled of having $X$, $p(s \in A | X, I) = 0.838$ just because your sample size compare to the rest of the dataset is relatively small. I'd however suggest to calculate $p(X|I)$ use all the information you have for the rest of the classes, then your calculation would be more acceptable.

I hope it was helpful :D

$\endgroup$
2
  • $\begingroup$ Excellent answer - when I can +1 I will. Would this technique be called a Naive Bayes Classifier? $\endgroup$ Commented Feb 6, 2014 at 12:07
  • $\begingroup$ no, here I've calculated the probability that an item with a single feature $X$ belongs to class $A$. However, if you have more than one feature, which is the case most of time, then the probability for being in class $A$ having 4 features $X, Y,Z, W$, looks like this: $p(s\in A| X, Y, Z, W)$. In general calculating this posterior needs information about the dependencies between different features, but this is sometimes quite impossible. Therefore, people came up with Naive Bayes method where the posterior factorizes: $p(s\in A| X,Y,Z,W) = p(s \in A| X)p(s \in A| Y)p(s \in A| Z)p(s \in A| W)$ $\endgroup$
    – omidi
    Commented Feb 6, 2014 at 12:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.