3
$\begingroup$

I'm writing a report of a machine learning system I built. The best F-score measures are similar or greater than another ML approaches in the state of art. However, my advisor outlines that I can't simply compare the F-score results of my system with the results of another systems in the state of art, because they don't use the same dataset I use. My question is how can I compare my results to another systems, provided that I don't have access to them, and so, I can just get values like accuracy, precision and recall from them?

Thanking you very much,

Improve this question
$\endgroup$
5
  • 3
    $\begingroup$ Most machine learning results are benchmarked against well-known and publicly available data sets, for example, MNIST or CIFRA are image recognition benchmarks. You should identify a commonly used benchmark appropriate for your particular method. $\endgroup$
    – Sycorax ♦
    Dec 31, 2018 at 4:25
  • $\begingroup$ Agreed, testing against a popular benchmark dataset is the best way to confirm your results. What type of problem are you working on, or what kind of data? $\endgroup$
    – Alex L
    Dec 31, 2018 at 4:32
  • $\begingroup$ I'm working on Android security, analyzing the application's permissions. In this field there is not that much of consensus about datasets, at least, through the papers I've read $\endgroup$
    – Sebastián Landínez García
    Dec 31, 2018 at 4:37
  • $\begingroup$ @SebastiánLandínezGarcía Have you seen the Android Malware Dataset? amd.arguslab.org $\endgroup$
    – Sycorax ♦
    Jan 2, 2019 at 1:12
  • $\begingroup$ Also, opendata.stackexchange.com is a forum for folks seeking specific data sets; review the forum's help center to understand more about what is and isn't on-topic there. $\endgroup$
    – Sycorax ♦
    Jan 2, 2019 at 2:01

1 Answer 1

Reset to default
1
$\begingroup$

Go for benchmark datasets as described in the answers.

Caveat: When you are comparing your results with one of the works in the literature, make sure to compare under similar settings (for example, same features, identical train_test split, same data preprocessing etc.)

If these change, then you cannot directly compare your results with the results available in the literature. In this case, you have to report your results along with the experiment settings.

Maybe that's what your supervisor implied.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.