I am doing my final thesis in the field of Deepfakes and their detection. The final outcome is to have a binary classifier which could predict which video was updated and which was not. In other words, if video is fake, output number close to 1 and if video is real, output number close to 0. One of the reasons I started this project is because Facebook released a massive dataset called DFDC and they announced in media that the best solution got only 82% average precision on validation dataset and 65% average precision on the testing dataset. Source: https://ai.facebook.com/datasets/dfdc/

However, they also released a research paper describing the results: https://arxiv.org/pdf/2006.07397.pdf This is the part where I am confused. Research paper does not mention anything about 82% and 65%. Contradictory, in their research paper's figure you can see that Average Precision of the best score was around 0.98 (on the public testing dataset)

enter image description here

However, they also include the log loss scores and I managed to understand how Facebook calculates the values of 82% and 65% on public dataset and private dataset. What they do is described in the answer here: What's considered a good log loss? which basically if the log-loss is 0.42, they take average precision as:

$$ ap = \frac{1}{{e^{0.42}}} = 0.657 $$

I was wondering if it is acceptable to use the above method because when I evaluate my score with scikit library and use: average_precision_score, I get much higher scores if to compare with the method which Facebook used as in What's considered a good log loss?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.