I'm writing a report of a machine learning system I built. The best F-score measures are similar or greater than another ML approaches in the state of art. However, my advisor outlines that I can't simply compare the F-score results of my system with the results of another systems in the state of art, because they don't use the same dataset I use. My question is how can I compare my results to another systems, provided that I don't have access to them, and so, I can just get values like accuracy, precision and recall from them?

Thanking you very much,

  • 3
    $\begingroup$ Most machine learning results are benchmarked against well-known and publicly available data sets, for example, MNIST or CIFRA are image recognition benchmarks. You should identify a commonly used benchmark appropriate for your particular method. $\endgroup$ – Sycorax says Reinstate Monica Dec 31 '18 at 4:25
  • $\begingroup$ Agreed, testing against a popular benchmark dataset is the best way to confirm your results. What type of problem are you working on, or what kind of data? $\endgroup$ – Alex L Dec 31 '18 at 4:32
  • $\begingroup$ I'm working on Android security, analyzing the application's permissions. In this field there is not that much of consensus about datasets, at least, through the papers I've read $\endgroup$ – Sebastián Landínez García Dec 31 '18 at 4:37
  • $\begingroup$ @SebastiánLandínezGarcía Have you seen the Android Malware Dataset? amd.arguslab.org $\endgroup$ – Sycorax says Reinstate Monica Jan 2 '19 at 1:12
  • $\begingroup$ Also, opendata.stackexchange.com is a forum for folks seeking specific data sets; review the forum's help center to understand more about what is and isn't on-topic there. $\endgroup$ – Sycorax says Reinstate Monica Jan 2 '19 at 2:01

Go for benchmark datasets as described in the answers.

Caveat: When you are comparing your results with one of the works in the literature, make sure to compare under similar settings (for example, same features, identical train_test split, same data preprocessing etc.)

If these change, then you cannot directly compare your results with the results available in the literature. In this case, you have to report your results along with the experiment settings.

Maybe that's what your supervisor implied.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.