# How to make a sequence element-wise clustering with a RNN (preferable in Keras)

Non-Keras contributions are also welcome since the question is very concrete already.

Imagine I have a sequence $$S_i = s_0, s_1, ..., s_n$$, where $$s_k$$ is the k-th element that represents an element of the sequence $$S_i$$. Now I want to cluster these sequence elements if they are "similar" (this similitude will be trained with training data).

Here is the structure of a training sample $$x_i$$ with len=6:

$$x_i = s_0, s_1, s_2, s_3, s_4, s_5$$

$$y_i = y_0, y_1, y_2, y_3, y_4, y_5$$

Where $$s_k$$ are vectors (300 dimensional vectors of Real values) and $$y_i$$ is the cluster where that vector pertains (e.g. 1).

Here is a possible example of the labels for this sample:

$$y_i = 0, 0, 1, 0, 1, 2$$

Meaning that:

• The elements 0, 1 and 3 of the sequence are related (therefore put in the same cluster)
• Elements 2 and 4 are also related, and
• Element 5 is put alone in another cluster.

I have 3 questions:

• How would this prediction be represented? As far as I know, I should return_sequences=True in the LSTM layer so for each input sequence, I have an output sequence, but I do not know what should I do with that output.
• What would my loss function be like? I guess it would be some kind of clustering metric like V-Measure Score, that gives an error wrt an input label of clusters, but I am not sure what would be the approach to implement a loss like this.
• What would the final layer of my network be like? Since given a sequence element, my prediction would have to be either to put the sequence element in a new cluster or some cluster where a prior sequence element has been put into, I am not sure how to model this.

I have found that this problem can simply be solved by encoding the input properly and targeting the correspondent sequences (also properly encoded).

Answering my own three questions: First of all, this is not a clustering problem since we have the labels for each sequence element. This is a supervised sequence labelling problem.

How would this prediction be represented? As far as I know, I should return_sequences=True in the LSTM layer so for each input sequence, I have an output sequence, but I do not know what should I do with that output.

Since you want to classify each sequence element, you should add a TimeDistributed Dense layer in order to make that Dense layer give a different output for each timestep. If the activation of that dense is the softmax function, then you have set up your network properly for a sequence labelling problem.

What would my loss function be like? I guess it would be some kind of clustering metric like V-Measure Score, that gives an error wrt an input label of clusters, but I am not sure what would be the approach to implement a loss like this.

V-Measure is to compare different clustering results of data and say how similar they are. Since here you are using it only to see if your model fits well, your inputs to the V-Measure would be (y_true, y_pred). Instead of doing that, your loss function can perfectly be the categorical crossentropy between your y_true and your y_pred, since you have the labels.

What would the final layer of my network be like? Since given a sequence element, my prediction would have to be either to put the sequence element in a new cluster or some cluster where a prior sequence element has been put into, I am not sure how to model this.

The output would be the TimeDistributed Dense with softmax activation we have seen before. The network itself will have learned if the input element should be put in a different class or not depending on the problem it has been trained to solve.

Here is a full example that solves my problem:

import keras
import random

letters = 'abc'

w_len = len(letters)

seq_len = 15
num_seqs = 1000

seqs = []

for i in range(num_seqs):
s = ''
for j in range(seq_len):
s += random.choice(letters)
seqs.append(s)

from keras.preprocessing.text import Tokenizer
tokenizer = Tokenizer(char_level=True, oov_token=None)
tokenizer.fit_on_texts(seqs)
seqs_keras = tokenizer.texts_to_sequences(seqs)

from keras import backend as K
import tensorflow as tf
with tf.Session() as sess:
seqs_keras_oh = K.one_hot(seqs_keras, w_len+1).eval()

def label_sample(x):
y = []
seen_idx = 0
d = {}
for i in range(len(x)):
if (x[i] not in d):
d[x[i]] = seen_idx
seen_idx += 1

y.append(d[x[i]])

return y

y = [label_sample(x) for x in seqs_keras]

with tf.Session() as sess:
y_oh = K.one_hot(y, w_len+1).eval()

from keras.layers import LSTM, Input, TimeDistributed, Dense
from keras.models import Model

encoded_length = seqs_keras_oh.shape

l_in = Input(shape=(seq_len, w_len+1))
l_lstm = LSTM(16, return_sequences=True)(l_in)
l_out = TimeDistributed(Dense(encoded_length, activation='softmax'))(l_lstm)

model = Model(inputs=l_in, outputs=l_out)

model.compile(
optimizer='rmsprop',
loss='categorical_crossentropy',
metrics=['accuracy']
)

model.fit(x=seqs_keras_oh, y=y_oh, batch_size=8, epochs=1000, validation_split=0.2)


After some epochs:

800/800 [==============================] - 1s 1ms/step - loss: 0.0195 - acc: 0.9988 - val_loss: 0.0180 - val_acc: 1.0000
Epoch 71/1000

• You asked three questions: could you explain how this post responds to them? – whuber Jun 13 at 21:35
• @whuber I have edited my own answer and added the answers to the questions. – Iván Sánchez Jun 15 at 9:36