Text mining of machine logs to find correlation between errors in R

I've with me 50 MB data from a machine consisting of event logs such as device status, warning and error. I wish to perform text mining on the same to find correlation between errors i.e. one error could trigger another in future and take a prescriptive action on it. I've used hierarchical clustering in R to generate a dendrogram but the result couldn't yeild expected insight. Hence, I've planned to perform Apriori Algorithm on the same. My queries are:

1. Is Hierarchical Clustering suggested to find such correlations?
2. Is Apriori an apt algorithm in such situation and why? Or, is there any other approach to solve this problem?

The logs that I've looks like this :

• You may get a better answer over at Cross Validated. Commented May 25, 2015 at 12:01

2 Answers

Apriori is one of the algorithms to solve Association rule learning a.k.a. Frequent Pattern Mining.

The problem is defined with sets of items in transactions. Within a single transaction, is there a set of items $A$ that will often tell us about with a second set of items $B$. First a minimum support is given by the user and is defined as the frequency of both $A$ and $B$ occurring in the same "transaction" over all transactions, i.e. $support=P(A \cup B)$. Next, the user defines a minimum confidence which is defined $confidence = P(B|A) = \frac{support(A \cup B)}{support(A)}$.

In this case, consider grouping the events from the logs over a period of time into a single transaction. Define the minimum support and minimum confidence, then Apriori will find which sets of events satisfy those minimums.

This usually returns many sets of items and usually a second metric is calculated to determine the best of these. Lift is the most common if these, but also consider these other association metrics

• Thanks for the response. The data that I have is machine logs, I'm skeptical as to how to convert it into transcational data. Having said that I performed Apriori on the log messages yielding me a confidence interval of 85% but I'm interested in finding correlation between the error messages and was wondering if there are other text mining techniques that could be helpful. Commented Jun 4, 2015 at 12:43
• @NeelimaSeshadri What problems are you having creating the transactional data? Is it difficult translating the log message to a event? Or are there problems grouping events into a transaction? Also, what does the confidence interval mean here? Commented Jun 6, 2015 at 19:54
• The logs that I've aren't sequential in order of occurence so I was wondering if conversion into transactional data would do the trick. I've attached a screenshot of the same in the question. Commented Jun 15, 2015 at 6:45

I think you should go for Apriori Algorithm

• This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post - you can always comment on your own posts, and once you have sufficient reputation you will be able to comment on any post.
– NathanOliver
Commented May 25, 2015 at 14:58
• @NathanOliver - Thank you, I've a query regarding posting questions if you could help me with it. I posted the same question in Cross Validated, is it incorrect to cross post the same question? Commented May 26, 2015 at 4:45