I build CNN network over documents with triplet loss. And compare documents with cosine similarity. It does really find similar docs and catches interesting dependencies. But simple tfidf model does it better in terms of my test set.
I use convolution over words (with word2vec embedding) in a sentence and then make second convolution over sentences. This gives me an unnormalized document vector, then I L2 normalize it.
Did you try to do something like this? Any suggestions?
It depends on your requirements and your data.
What do you mean by similarity? Do you mean similarity by word distribution or categorical similarity? I thought you have some labeled data but I am not sure what kind of labels you have. Let's say you have lots of documents from different categories and we can treat all pairs from the same categories as positive pairs and from different categories as negative pairs, then we can be sure that the distance between different categories will be enlarged and hence it can help for document similarity search.
You mentioned your encoder, two layers of CNN, but not its structure. Most probably your model suffers from the high bias problem because it performs worse even than TFIDF. I would recommend you check the training loss to see if it really gets stuck by training cases. If so you can use the now popular BERT and use its pretrained weights(transfer learning for regularization). A two layer CNN model would be expressive enough for the task with your size of data depending on the filters and kernel size you use. You can use reduce the regularization by reduce the dropout rate and etc to make the bias smaller. Or maybe the variance would be large and try to fix that.
The safe way is to train your model using two loss functions(multi-task learning) and interleave the two training process by setting a condition according to each other's loss. That's you combine the original loss(task) and the triplet loss to make sure that you can make some gains from the triplet loss. You can design the strategy.
I tried to use denoising autoencoder with triplet loss to generate document's embeddings and apply cosine similarity on it.
I tested both approach (tfidf and embeddings) on a news articles dataset. And embeddings approach gives better result in finding new articles of same category (i.e. Internalional, Sports, etc). While tfidf gives better result in finding new articles of same event (i.e. USA president selection, etc)
So you may take different approach depends on what you need.
