I am looking for methods to calculate a reasonable similarity metric or clustering for my data. I have data in the form of a number of distinct events (lets call them A, B, C, D, ...), and the time between the events. A few example data points could look like,
X1 = [(A, 0), (B, 2.2), (C, 3.2), (A, 4.0)]
X2 = [(B, 0), (C, 2.2)]
X3 = [(B, 0), (C, 2.2), (D, 4.2)]
X4 = [(B, 0), (A, 2.2), (E, 1.5)]
X5 = [(B, 0), (F, 2.2), (E, 1.5), (F, 5.0), (G, 2.0), (H, 2.0)]
X6 = [(B, 0), (A, 0.0), (E, 1.5), (F, 5.0), (G, 2.0)]
X7 = [(I, 0), (J, 0.0), (K, 1.5), (L, 5.0), (M, 2.0)]
To explain the format: each entry in the array is in the format event, time since first event. E.g. for X1, at time=0, event A happened. At time=3.2 (after event A), event C happened, and so forth. Also note that multiple events may happen at the same time. In reality, the sequences are typically much longer (50-100ish). The "vocabulary" of events is 1500ish. The time between two events ranges from 0.0-100.
I have a good few million data points, give or take, so I have been trying to look into a clustering based approach. From the above example, I would expect X1, X2, and X3 to have some similarity (X4 maybe, since the pattern A->B is just reversed). Also, X5 and X6 should be fairly similar. X7 should not have any similarity to any of the patterns. This is a somewhat fuzzy explanation, and that is intended. I am not looking for the one method that clusters perfectly - I am just looking for one that does an OK job taking all variables into consideration
I have been looking into things like Latent Dirichlet Allocation, Self Organising Maps, unsupervised Recurrent Neural Networks, Autoencoders, just counting the similarity between triplets, ...
My major problems in finding something reasonable has been,
- The method expects an equal number of features
- The method sees the sequence as a bag-of-words
- The method does not implicitly handle a mixture of categorical and continous data
- The method has no notion of time
I put some very broad categories on this question, simply because I am looking for suggestions or pointers to software/papers.
Any help would be much appreciated, and please do comment if something is not clear.
Edit 1: I started looking into Recurrent Neural Networks, but the information is quite dense, and the unsupervised part does not seem to be that well developed. The best paper I could find was Klapper-Rybicka et al. (2001), but they apply information theoretic approaches tailored specifically for continous acoustic data (Binary Information Gain Optimization and Nonparametric Entropy Optimization), and I am not quite sure how to translate these measures to the type of data that I have described above.
If anyone could elaborate on the unsupervised training of recurrent neural networks using data as I have described above I would be very thankful.