38
$\begingroup$

I'm getting a 100% accuracy for my decision tree. What am I doing wrong?

This is my code:

import pandas as pd
import json
import numpy as np
import sklearn
import matplotlib.pyplot as plt


data = np.loadtxt("/Users/Nadjla/Downloads/allInteractionsnum.csv", delimiter=',')


x = data[0:14]
y = data[-1]


from sklearn.cross_validation import train_test_split

x_train = x[0:2635]
x_test = x[0:658]
y_train = y[0:2635]
y_test = y[0:658]


from sklearn.tree import DecisionTreeClassifier
tree = DecisionTreeClassifier()
tree.fit(x_train.astype(int), y_train.astype(int))


from sklearn.metrics import accuracy_score

y_predicted = tree.predict(x_test.astype(int))
accuracy_score(y_test.astype(int), y_predicted)
$\endgroup$
  • 4
    $\begingroup$ Why do you think you are doing something wrong? Perhaps your data are such that you can achieve a perfect classication... $\endgroup$ – Knarpie Mar 22 '18 at 12:39
  • 64
    $\begingroup$ Incidentally, +1 for wondering whether something is wrong with 100% accuracy. Far too many people would just think their model is great... $\endgroup$ – Stephan Kolassa Mar 22 '18 at 14:36
  • 1
    $\begingroup$ In R there's a package (caret) to automatically split a dataset into two groups, one for training data and the other one for testing data. I call the process as the data partition. I believe there's a similar package in Python to achieve a data partition as well. $\endgroup$ – Anastasiya-Romanova 秀 Mar 23 '18 at 1:48
  • $\begingroup$ Useful background reading: Common Pitfalls in ML $\endgroup$ – smci Mar 23 '18 at 3:07
  • 3
    $\begingroup$ @Anastasiya-Romanova秀 Pretty much every serious ML library contains this functionality, including the one used by OP (OP even imported the relevant functionality, and just didn’t use it for some reason). $\endgroup$ – Konrad Rudolph Mar 23 '18 at 15:17
79
$\begingroup$

Your test sample is a subset of your training sample:

x_train = x[0:2635]
x_test = x[0:658]
y_train = y[0:2635]
y_test = y[0:658]

This means that you evaluate your model on a part of your training data, i.e., you are doing in-sample evaluation. In-sample accuracy is a notoriously poor indicator to out-of-sample accuracy, and maximizing in-sample accuracy can lead to overfitting. Therefore, one should always evaluate a model on a true holdout sample that is completely independent of the training data.

Make sure your training and your testing data are disjoint, e.g.,

x_train = x[659:2635]
x_test = x[0:658]
y_train = y[659:2635]
y_test = y[0:658]
$\endgroup$
  • 2
    $\begingroup$ It would be better to use sklearn.model_selection.train_test_split as Juan Ignacio Gil suggests since this also shuffles the sets and avoids concerns if the dataset is non-random in ordering. It's also clearer because it shows intent, and automatically handles changes in the size of dataset. $\endgroup$ – Jack Aidley Mar 26 '18 at 8:32
  • 1
    $\begingroup$ @JackAidley: I agree (and upvoted Juan's answer some days ago). Even better would be to make the split deterministic for debugging by setting the random number seed. $\endgroup$ – Stephan Kolassa Mar 26 '18 at 9:25
  • $\begingroup$ @StephanKolassa Hi, I've been tweaking with the Iris dataset, and after using GridSearchCV with training data, for testing accuracy I get 100% with KNeighborsClassifier. I have used test_train_split for splitting the dataset. What could I have done wrong here? $\endgroup$ – Sndn Aug 13 '18 at 12:20
19
$\begingroup$

You are getting 100% accuracy because you are using a part of training data for testing. At the time of training, decision tree gained the knowledge about that data, and now if you give same data to predict it will give exactly same value. That's why decision tree producing correct results every time.

For any machine learning problem, training and test dataset should be separated. Accuracy of the model can be determined only when we examine how it is predicting for unknown values.

$\endgroup$
17
$\begingroup$

As other users have told you, you are using as test set a subset of the train set, and a decision tree is very prone to overfitting.

You almost had it when you imported

from sklearn.cross_validation import train_test_split

But then you don't use the function. You should have done:

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.33)

to get random train and test sets

$\endgroup$
2
$\begingroup$

As pointed by @Stephan Kolassa and @Sanjay Chandlekar, this is due to the fact that your test sample is a subset of your training sample.

However, for the selection of those samples, random sampling would be more appropriate to ensure that both samples are representative. Depending on your data structure, you might also consider stratified random sampling.

I'm not fluent in Python but any statistical software should allow random sampling; some hints are also available on SO.

$\endgroup$
0
$\begingroup$

Just want to chime in on the intuition for why you need to split training and test samples explicitly.

If you have $n$ observations and make $n$ (actually, $n-1$, and possibly far less) splits on your data, you will perfectly classify every point (if this isn't immediately clear, write down some small-scale examples, e.g., $n = 2$, and convince yourself of this).

This is called overfitting because this splitting process is exceedingly unlikely to be predictive of data points that are relevant to your problem but which you haven't yet observed.

Of course the whole point of building these prediction platforms is to create tools which can be applied to never-before-seen data; splitting the data we have into training and test samples is an attempt to simulate this self-blinding and police our models from overfitting in the above fashion.

$\endgroup$
0
$\begingroup$

You don't need 100% accuracy to get overfitting. With enough buckets, you can get irreproducible results (something that would look terrible out-of-sample).

See this excerpted article from the Lancet, describing the method of chopping a sample into buckets which are far too fine. Munchausen's Statistical Grid It is also the basis for the XKCD cartoon Significant

Achieving 100% accuracy is just a short step away from finding a classifier which works deceptively well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.