I am currently working on paraphrase detection using the gensim doc2vec utilities. Basically I map sentences to fixed-length vectors using doc2vec and then calculate the cosine-similarity on them. I found many papers on doc2vec and just cannot manage to reproduce their results. Since none of the papers describes how exactly they find their threshold to calculate the F1 score, I assume that I am doing this step wrong.
Currently I do this:
- Train doc2vec on english wikipedia in an unsupervised way (tried with/without lemmatization)
- Evaluate the model on the MSRP dataset (about 5000 labeled paraphrases/non-paraphrases). I only use the training part of the dataset in this step.
- On the resulting ROC curve, I find the optimal threshold by determining which threshold minimizes abs(FPR - FNR). Effectively I am looking for the threshold that comes closest to the EER (equal error rate). I am not sure if this is the way to go.
- Using the EER threshold, I evaluate the model again on the test part of the MSRP dataset. Based on this, I calculate accuracy, precision, recall and the F1 score that is found in many papers related to paraphrase detection.
Unfortunately the best F1 score that I could achieve with any of my models is about 5% worse than the worst baseline in most papers. I do not manage to get a F1 score above 70%, many papers achieve nearly 80% in their best configuration.
So my question is: Is using the EER threshold a good idea here? I chose this point because I think that it is a good compromise to have both types of error equally low. Or should I try to do a grid search on the possible thresholds in order to find the threshold which maximizes the F1 score?
Maybe someone here has more experience and can tell me how to calculate useful thresholds.
EDIT: I just changed my code to test a few different thresholds and surprisingly I get much higher F1 scores when I choose lower thresholds. Here are my results:
T: 0.000000 F1: 0.798747
T: 0.100000 F1: 0.798747
T: 0.200000 F1: 0.798747
T: 0.300000 F1: 0.798747
T: 0.400000 F1: 0.798747
T: 0.500000 F1: 0.798747
T: 0.600000 F1: 0.798747
T: 0.700000 F1: 0.798184
T: 0.800000 F1: 0.800738
T: 0.900000 F1: 0.659753
T: 1.000000 F1: 0.000000
(The EER threshold usually was around 0.85, so very high)
Anyways, when looking at the values, it seems strange to me. The F1 score does not change before 0.7. Could the reason for this be that my classificator generally outputs high values and that the distribution of true and false samples in the data set is not equal, i.e. there are more positives (1147) than negatives (578) in the test split?
I also have a low very high recall (100%) and a low precision (66%) when I use for example threshold=0.5. Those 66% perfectly match the ratio of positives to total samples in the data set. So to sum it up, when setting threshold = 0.5, I simply classify all test samples as paraphrases and since the data set is not equally distributed, this gives me a very high recall and a low precision.
If I move the threshold to > 0.9, I get a much higher precision and lower recall.
To me this feels like cheating. I am not sure if the authors of the many papers really find their thresholds like this.
Anyone has an idea?
Here is a plot of different metrics using all thresholds between 0 and 0.1 (0.05 steps):