How to make a sequence element-wise clustering with a RNN (preferable in Keras)

Non-Keras contributions are also welcome since the question is very concrete already.

Imagine I have a sequence $$S_i = s_0, s_1, ..., s_n$$, where $$s_k$$ is the k-th element that represents an element of the sequence $$S_i$$. Now I want to cluster these sequence elements if they are "similar" (this similitude will be trained with training data).

Here is the structure of a training sample $$x_i$$ with len=6:

$$x_i = s_0, s_1, s_2, s_3, s_4, s_5$$

$$y_i = y_0, y_1, y_2, y_3, y_4, y_5$$

Where $$s_k$$ are vectors (300 dimensional vectors of Real values) and $$y_i$$ is the cluster where that vector pertains (e.g. 1).

Here is a possible example of the labels for this sample:

$$y_i = 0, 0, 1, 0, 1, 2$$

Meaning that:

• The elements 0, 1 and 3 of the sequence are related (therefore put in the same cluster)
• Elements 2 and 4 are also related, and
• Element 5 is put alone in another cluster.

I have 3 questions:

• How would this prediction be represented? As far as I know, I should return_sequences=True in the LSTM layer so for each input sequence, I have an output sequence, but I do not know what should I do with that output.
• What would my loss function be like? I guess it would be some kind of clustering metric like V-Measure Score, that gives an error wrt an input label of clusters, but I am not sure what would be the approach to implement a loss like this.
• What would the final layer of my network be like? Since given a sequence element, my prediction would have to be either to put the sequence element in a new cluster or some cluster where a prior sequence element has been put into, I am not sure how to model this.