2
$\begingroup$

I have two data-sets for same samples. But they are produced using two different instruments. I want to choose one data-set for further analysis. How can I find/prove which data-set is better?

To make it more simple - let's say we took photos of 50 males and 50 females using two different types of camera, and produce two different data-sets. Both data-sets will have same dimension but the data-values will be different. We want to use one of these data-sets for machine-learning (classification). But the first step will be to choose the better data-set. How to do that?

Note 1- As some people suggested simplest way will be to apply machine-learning method on both data-sets and do cross-validation to see which data-set gives better accuracy. This is exactly what I thought in the beginning. Unfortunately this is not a feasible path in case of my data-sets primarily because of the fact that I have several (>1000) data-sets and every data-set have very high dimension with many classes. So I am looking for some kind of statistics which can give some indication about quality of data which will allow to pick few data-sets for further analysis.

Note 2 - In this case quality of data means 'In a data-set how well separated are classes compare to other data-set'.

$\endgroup$
1
  • 1
    $\begingroup$ The data-sets are going to be used for supervised classification? are they labeled? If so, you could initilly just separate them in an 80% training set / 20% test set. Train your model on the training sets and see how it predicts on the test sets. See which one yields more precise results. Or do some form of cross validation for the same purpose. $\endgroup$
    – JEquihua
    Commented Mar 12, 2013 at 16:39

1 Answer 1

2
$\begingroup$

What do you exactly mean by "better data set"?

I would understand it as: with which data set can I train a classifier that yields best results?. If so, I would train two classifiers, say SVM, one for each of the data sets. For each one I would choose the parameters as to give best performance for that data set in terms of cross-validation.

Then I would see how each classifier performs on the other data set. If one classifier has a better performance on BOTH sets, I would use this result as a criteria to pick up the data set on which that classifier was trained. Notice that the result depends on the particular classifier chosen.

You could try this with several different classifiers, like kNN, MLPs, or anything else.

Another question would be: which classifier performs (generalize) better across different data sets. For that please refer to the paper: "Statistical Comparison of Classifiers over Multiple Data Sets", Janez Demsar, JMLR

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.