4
$\begingroup$

I've with me 50 MB data from a machine consisting of event logs such as device status, warning and error. I wish to perform text mining on the same to find correlation between errors i.e. one error could trigger another in future and take a prescriptive action on it. I've used hierarchical clustering in R to generate a dendrogram but the result couldn't yeild expected insight. Hence, I've planned to perform Apriori Algorithm on the same. My queries are:

  1. Is Hierarchical Clustering suggested to find such correlations?
  2. Is Apriori an apt algorithm in such situation and why? Or, is there any other approach to solve this problem?

The logs that I've looks like this :

enter image description here

$\endgroup$

migrated from stackoverflow.com May 25 '15 at 20:07

This question came from our site for professional and enthusiast programmers.

1
$\begingroup$

Apriori is one of the algorithms to solve Association rule learning a.k.a. Frequent Pattern Mining.

The problem is defined with sets of items in transactions. Within a single transaction, is there a set of items $A$ that will often tell us about with a second set of items $B$. First a minimum support is given by the user and is defined as the frequency of both $A$ and $B$ occurring in the same "transaction" over all transactions, i.e. $support=P(A \cup B)$. Next, the user defines a minimum confidence which is defined $confidence = P(B|A) = \frac{support(A \cup B)}{support(A)}$.

In this case, consider grouping the events from the logs over a period of time into a single transaction. Define the minimum support and minimum confidence, then Apriori will find which sets of events satisfy those minimums.

This usually returns many sets of items and usually a second metric is calculated to determine the best of these. Lift is the most common if these, but also consider these other association metrics

$\endgroup$
  • $\begingroup$ Thanks for the response. The data that I have is machine logs, I'm skeptical as to how to convert it into transcational data. Having said that I performed Apriori on the log messages yielding me a confidence interval of 85% but I'm interested in finding correlation between the error messages and was wondering if there are other text mining techniques that could be helpful. $\endgroup$ – Naive Babes Jun 4 '15 at 12:43
  • $\begingroup$ @NeelimaSeshadri What problems are you having creating the transactional data? Is it difficult translating the log message to a event? Or are there problems grouping events into a transaction? Also, what does the confidence interval mean here? $\endgroup$ – Eric Farng Jun 6 '15 at 19:54
  • $\begingroup$ The logs that I've aren't sequential in order of occurence so I was wondering if conversion into transactional data would do the trick. I've attached a screenshot of the same in the question. $\endgroup$ – Naive Babes Jun 15 '15 at 6:45
0
$\begingroup$

I think you should go for Apriori Algorithm

$\endgroup$
  • 2
    $\begingroup$ This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post - you can always comment on your own posts, and once you have sufficient reputation you will be able to comment on any post. $\endgroup$ – NathanOliver May 25 '15 at 14:58
  • $\begingroup$ @NathanOliver - Thank you, I've a query regarding posting questions if you could help me with it. I posted the same question in Cross Validated, is it incorrect to cross post the same question? $\endgroup$ – Naive Babes May 26 '15 at 4:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.