24
$\begingroup$

I was impressed by the results in the ICML 2014 paper "Distributed Representations of Sentences and Documents" by Le and Mikolov. The technique they describe, called "paragraph vectors", learns unsupervised representations of arbitrarily-long paragraphs/documents, based on an extension of the word2vec model. The paper reports state-of-the-art performance on sentiment analysis using this technique.

I was hoping to evaluate this technique on other text classification problems, as an alternative to the traditional bag-of-words representation. However, I ran across a post by the second author in a thread in the word2vec Google group that gave me pause:

I tried myself to reproduce Quoc's results during the summer; I could get error rates on the IMDB dataset to around 9.4% - 10% (depending on how good the text normalization was). However, I could not get anywhere close to what Quoc reported in the paper (7.4% error, that's a huge difference) ... Of course we also asked Quoc about the code; he promised to publish it but so far nothing has happened. ... I am starting to think that Quoc's results are actually not reproducible.

Has anyone had success reproducing these results yet?

$\endgroup$
2
  • $\begingroup$ Has this situation changed, yet? I know that Gensim has implemented a version of doc2vec (paragraph/document vectors), see: radimrehurek.com/gensim/models/doc2vec.html but no attempt to reproduce the results in the paper cited here. $\endgroup$ Mar 2, 2015 at 20:44
  • 1
    $\begingroup$ Yes, there were attempts to reproduce the paper results using gensim: see the doc2vec IPython notebook. $\endgroup$
    – Radim
    Aug 24, 2015 at 3:08

1 Answer 1

17
$\begingroup$

Footnote of the paper "Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews" (one of the authors is Tomas Mikolov) says:

In our experiments, to match the results from (Le & Mikolov, 2014), we followed the suggestion by Quoc Le to use hierarchical softmax instead of negative sampling. However, this produces the 92.6% accuracy result only when the training and test data are not shuffled. Thus, we consider this result to be invalid.

$\endgroup$
2
  • 1
    $\begingroup$ I don't understand why "not shuffled" ==> invalid. Is there no well-defined split between train/test set? So that what is train/test depends on how you shuffle the (original) dataset? The order of the test set shouldn't matter (there's no dynamic evaluation, right?). And the order of the training set shouldn't matter much, either... $\endgroup$ Oct 17, 2016 at 21:22
  • $\begingroup$ @user2429920 If they're getting differences, then clearly the order does matter somehow. $\endgroup$
    – JAB
    Nov 28, 2017 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.