3
$\begingroup$

I would like to do some sentence embedding on around 500 sentences. The purpose is to find for new sentences, the most similar ones within the 500 sentences. Unfortunately, for now its definitely not working. Indeed, to test my model I simply looked at the 10 most similar sentences for a sentence which was used for training, hopping that the most similar one will be itself. But its not the case. Hence, I am wondering how to improve such models (e.g. the only way I know is shuffling the data)? Also, is there other ways to test a doc2vec model?

I dont know how to share the data, knowing that I dont have dropbox or similar, perhaps by email...

Thank you very much for any suggestion.

Here is my code:

import numpy as np
import re as re
import gensim
from gensim.models import doc2vec
from collections import namedtuple
#for stopwords
from nltk.corpus import stopwords
import nltk
nltk.download('stopwords')
from random import shuffle

#import data
li_doc = list(np.load('/home/camille/vm_exchange/li_doc.npy'))


def text_cleaning_for_embeding(x):
    #remove ponctuation
    x = ' '.join(re.split('[?.,:/>!;]', x))
    #remove entre parenthèse
    x = x.replace('(','')
    x = x.replace(')','')
    #remove digits
    x = ''.join([l.lower() for l in x if not l.isdigit()])
    #remove stopwords and tokenize
    x = ' '.join([w for w in x.split() if w not in cachedStopWords])
    return(x)

#Transform data to 'TaggedDocument'
docs = []
analyzedDocument = namedtuple('AnalyzedDocument', 'words tags')
cachedStopWords = stopwords.words("english")

for i, text in enumerate(li_doc):
    text_ = text_cleaning_for_embeding(text)
    #list of words
    words = text_.split()
    #small cleaning
    words = [i.lower().strip() for i in words]
    #register as namedtuple: analyzedDocument
    tags = [i]
    docs.append(analyzedDocument(words, tags))



docs_train = docs[0:500]
docs_test = docs[500:len(docs)]


model = gensim.models.Doc2Vec(vector_size=100, window=10, min_count=1, workers=8, alpha=0.025, min_alpha=0.015, 
                              epochs=20)
#sample=1e-4, negative=5,
#shuffling is better (ot needed at each trianing epoch
shuffle(docs_train)
#Build vocabulary from a sequence of sentences 
model.build_vocab(docs_train)
#Update the model’s neural weights from a sequence of sentences
model.train(docs_train, epochs=model.epochs, total_examples=model.corpus_count)



#testing on 2 training sentences:
for i in range(0,2):
    t = docs_train[i].tags[0]
    new_vector = model.infer_vector(docs_train[i].words,steps=50, alpha=0.025)
    sims = model.docvecs.most_similar([new_vector]) #gives you top 10 document tags and their cosine similarity
    #model.docvecs.most_similar(): get similarity between word vector
    #model.most_similar(): get similarity between document
    print('--------------------------------------------------------------------------------------------------------')
    print('------------- %s'%li_doc[t])
    print('--------------------------------------------------------------------------------------------------------')
    for t,sim in sims:
        print('\n', sim, li_doc[t])

the output:


------------- Apple cider (also called sweet cider or soft cider or simply cider) is the name used in the United States and parts of Canada for an unfiltered, unsweetened, non-alcoholic beverage made from apples.

0.9132258296012878 Chocolate ice cream is ice cream with natural or artificial chocolate flavoring. Chocolate is the second most common flavor of ice cream in the United States, after vanilla.

0.9112349152565002 Spice cake is traditionally flavored with a mixture of spices. The cake can be prepared in many varieties.

0.9098634719848633 Vanilla is frequently used to flavor ice cream, especially in North America and Europe. Vanilla ice cream, like other flavors of ice cream, was originally created by cooling a mixture made of cream, sugar, and vanilla above a container of ice and salt.

0.9087004661560059 Ice cream (derived from earlier iced cream or cream ice) is a sweetened frozen food typically eaten as a snack or dessert.

0.9066498279571533 Hot chocolate, also known as Chocolate tea, drinking chocolate or just cocoa is a heated beverage consisting of shaved chocolate, melted chocolate or cocoa powder, heated milk or water, and usually a sweetener.

0.906596839427948 Multigrain bread is a type of bread prepared with two or more types of grain. Grains used include barley, flax, millet, oats, wheat, and whole-wheat flour, among others.

0.9057920575141907 Cream of mushroom soup is a simple type of soup where a basic roux is thinned with cream or milk and then mushrooms and/or mushroom broth are added.

0.9056931138038635 A plum tomato, also known as a processing tomato or paste tomato, is a type of tomato bred for sauce and packing purposes.

0.9055744409561157 A meatball is ground meat rolled into a small ball, sometimes along with other ingredients, such as bread crumbs, minced onion, eggs, butter, and seasoning.

0.9055196642875671 Fermented milk products, also known as cultured dairy foods, cultured dairy products, or cultured milk products, are dairy foods that have been fermented with lactic acid bacteria such as Lactobacillus, Lactococcus, and Leuconostoc.


------------- Salami (singular salame) is a type of cured sausage consisting of fermented and air-dried meat, typically beef or pork.

0.873619556427002 A sausage is a cylindrical meat product usually made from ground meat, often pork, beef, or veal, along with salt, spices and other flavourings, and breadcrumbs, encased by a skin.

0.8629544973373413 Cornbread is any quick bread containing cornmeal. They are usually leavened by baking powder.

0.8617033958435059 Sauerkraut (; German: [ˈzaʊɐˌkʁaʊt] ( listen)) is finely cut cabbage that has been fermented by various lactic acid bacteria.

0.8616538643836975 Whisky or whiskey is a type of distilled alcoholic beverage made from fermented grain mash. Various grains (which may be malted) are used for different varieties, including barley, corn (maize), rye, and wheat.

0.8616343140602112 Almond butter is a food paste made from almonds. Almond butter may be crunchy or smooth, and is generally "stir" (susceptible to oil separation) or "no-stir" (emulsified).

0.8612356185913086 A flour tortilla (, ; or wheat tortilla to differentiate it from other uses of the word tortilla, which in Spanish means "small torta", or "small cake") is a type of soft, thin flatbread made from finely ground wheat flour from Mexico.

0.8611332178115845 Toffee is a confection made by caramelizing sugar or molasses (creating inverted sugar) along with butter, and occasionally flour.

0.8610053658485413 Semolina is the coarse, purified wheat middlings of durum wheat mainly used in making pasta and couscous.

0.8609529733657837 White chocolate is a chocolate derivative. It commonly consists of cocoa butter, sugar and milk solids and is characterized by a pale yellow or ivory appearance.

0.8609246611595154 A hamburger, beefburger or burger is a sandwich consisting of one or more cooked patties of ground meat, usually beef, placed inside a sliced bread roll or bun.

$\endgroup$

1 Answer 1

3
$\begingroup$

I can provide a few suggestions if you're still looking for help (as of this writing, the question is ≈1 year old)

  • First thing I noticed was the number of epochs you used while training epochs=20 could be a bit low, especially considering that your documents are shorter in length. This is a little bit trial-and-error as I'm not as familiar with training on shorter documents, but have you tried bumping up epochs higher, perhaps something within the 50-100 range? (maybe even 100+) This may perform better on shorter documents.

  • Your number of documents (500) is low in count - if possible, could you gather more examples for training?

As a side note. I've noticed that you perform the document list slicing first, then shuffle docs_train for training. Not sure if that's intended: Try shuffling first and then slice. Also, it depends on why exactly you'd want to arbitrarily train on a portion of your documents - If the goal is to find similar documents going forward, I'd also try training your doc2vec model on the entire set, to find similar documents within the training set. Then, when you have an out-of-sample document, you can infer the vector to find most similar sentences within what the model knows and trained on previously.

In practice, I've done a "similarity search" by inferring all new documents (out-of-sample) and performing a cross join to surface up documents that are most similar, where my training was the entire set.

  • During the call to model.infer_vector(), you could also try bumping up steps=50 (btw, steps arg is deprecated, you could switch to use epochs=N) I've found the optimal number under my use cases was 99-250 range, but this requires some tuning and testing on your end, as that number may not be the same for you. How I've tackled this was to perform iterative inference (for epochs=10, 20, 30... 300+) and chart the performance of cosine similarity between the model-known vector vs the inferred vector.

  • You could try tweaking the vector_size from 100 to something smaller or larger. If you think the set of vocabulary is limited, perhaps condensing vectors to smaller size may achieve better results because you are representing vocab by mapping to smaller vector size.

  • for short corpuses, you could try word stemming, as this may help the model. Likewise, try options for keeping stopwords and punctuation.

  • If you are inclined, you can try more hyperparam tuning/testing: set dm=0, window={3-10}, min_count={1-5}, etc... if you package up your testing code you try "grid search" by testing various doc2vec configurations and see which one comes out with the best accuracy.

Anyway, I hope this helps to at least get you started on right path on what to experiment with and tweak in your code! I've had to learn a bunch just via testing and lots of reading other posts/questions&answers - In reality, the one big takeaway I've found is that your doc2vec implementation greatly depends on your use-case.

For more reading, I've found this paper very useful to understand underlying mechanics for doc2vec: An Empirical Evaluation of doc2vec with Practical Insights into Document Embedding Generation

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.