I'm a CS master student focusing on data mining. Now I'm doing my master thesis and the contribution of my thesis is to compare different approaches/methods of one topic (e.g. clustering of text documents). What I did so far is looking at the state of the art on the topic and read the papers. However now I need to think of what to compare. Of course there is the obvious comparison question: which method give best results? But that is so obvious. My supervisor once mentioned to see how those methods compare their results and look that maybe there is a better way to compare the results of the methods. That was helpful to me to think about questions like this. But still I'm so stuck and I can only think of obvious stuff. This is the first time I do something like that.

I'm sure some of you went through something like this, so I was wondering if you can even tell me some /basis/ stuff and questions that people address when they compare methods in computer science. My main questions to you are:

1- When I read the papers of the methods to compare, in which way I should read it? Critical? Questionable? What to look exactly in them?

2- Is there a general scheme for comparing methods in academia?

3- As for a master thesis, what which stuff are a must-do for comparisions?

4- Any great references/papers related to comparisions that could help me?


2nd point. All methods both in and out of academia are intended to solve practical problems. There is no general scheme, rather the performance is compared according to the task in hand. Most common criteria are

  • predictive power
  • interpretability (more important for clustering, for there are no loss function in general)

but there might be another ones, such as

  • scalability to large datasets
  • robustness to outliers
  • handling missing values
  • ability to enrich feature set with linear combinations of original features

and so on, depending on the problem formulation and data peculiarities.

3rd point. If you deal with supervised learning, you probably should tune parameters via cross-validation, and compare methods on a test set.

4th point. Magnificent free book (somewhat technical, but not too much): Elements of Statistical Learning http://statweb.stanford.edu/~tibs/ElemStatLearn/


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.