I have a dataset with 1175 examples and 21 features which are in the range of [-1, +1], and two class labels 1 and 0. As I read in the most of the resources, it is good to have data in the range of [-1, +1] or [0, 1]. So I thought I don't need any preprocessing. But when I run SVM and decision tree classifiers from scikit-learn, I got 100% accuracy using cross-validation with 10 folds. However the classification accuracy seems to decrease as I perform more iterations.

I am collecting these datas from a Kinect device which gives me angles and positions of the certain joints of the human body. Because of hardware faults I am sure that there are noise on data. So getting 100% is almost impossible.

My dataset is avaiable here

  • $\begingroup$ Are you sure there is no bugs in the code? The 100% accuracy on Cross Validation doesn't seem plausible. You may consider checking for overfitting, but again - you should see it on CV. Also: what are the classes? Maybe the classes are directly related to your features, so there is nothing to learn because the relation is "obvious"? $\endgroup$ – Tim May 8 '15 at 20:42
  • $\begingroup$ How is the label distribution over the data set? (How many of the 1175 examples are labeled with 0 / with 1) $\endgroup$ – Christopher Schröder May 8 '15 at 22:37
  • 1
    $\begingroup$ Sharing your data set alone is not enough. The preferred way to get help on StackExchange sites, such as Cross Validated and StackOverflow, is to create a minimal reproducible example (aka minimal working example). If you cannot do that, at least, provide essential parts of your code, which produces the result in question. $\endgroup$ – Aleksandr Blekh May 9 '15 at 2:00

I was able to reproduce your results:

> clf = svm.SVC()
> scores = cross_validation.cross_val_score(clf, X, Y, cv=10)

I didn't get perfect out of fold classification, but close:

> print(scores)
array([ 1.        ,  1.        ,  1.        ,  0.99152542,  1.        ,
        1.        ,  1.        ,  1.        ,  1.        ,  1.        ])

It's not very easy to figure out what's going on with a support vector machine, so I fit a decision tree to your data:

> tre = tree.DecisionTreeClassifier()
> tre.fit(X, Y)

The tree is a prefect classifier on the training data:

> sum(abs(tre.predict(X) - Y))

Turns out this tree is pretty simple:

enter image description here

It looks like the third column in your data (the one named Z) is a perfect separator. This is easily confirmed with a scatterplot:

enter image description here

  • $\begingroup$ I had the same tree with you which act lıke the maın decısıon maker. It seems that this dataset doesn't require a ML classıfıer at all. A simple if rule will do the job(wıth correct threshold), but it was good to understand this situation since I have plenty of this kind of dataset that I need to see if this situation is exist. So the way you did really help me. Thank you! $\endgroup$ – robowolf May 9 '15 at 15:46
  • $\begingroup$ Well, the classifier (the tree) still told you something useful about the shape of your data, and that's a big part of learning. Since you feel your question has been addressed, could you accept my answer? $\endgroup$ – Matthew Drury May 9 '15 at 16:50

Oh, my God, I came across the same issue. Maybe my answer is not the best answer for you, but it may help other people.

Here is my code with Scikit-Learn

clf = DecisionTreeClassifier(criterion='entropy', max_depth=10)
clf.fit(X, y)

And I got 100% accuracy score.

However, when I got the feature_importances_ of clf, and I found the tag column was in X which should be removed from X, after removing the tag column from X, the accuracy was 89%.

So, my suggestion is after you built a model, check the parameters in it, for example feature_importances_ etc. Good luck!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.