0
$\begingroup$

I've tested in-sample evaluation with different classifiers (Decision trees, Random Forests, Gaussian Naive Bayes) within sklearn and Iris datasets.

import sklearn.datasets as datasets
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import RandomForestClassifier


iris=datasets.load_iris()
df=pd.DataFrame(iris.data, columns=iris.feature_names)
y=iris.target

First with decision trees, I get 100% accuracy

clf=DecisionTreeClassifier()
clf.fit(df,y)
print(str(clf.score(df, y)))

For other classifiers, the score is strictly lower than 100%

For example within Random forests I have:

clf = GaussianNB()
clf.fit(df,y)
print(str(clf.score(df, y)))

In [47]: 0.96

With RandomForestClassifier, the score is 98%. It's also the case with other classifiers such as KNeighbors.

My question:

Is there a theoretical reason for which decision trees give a 100% accuracy score for in-sample evaluations, and why the other classifiers do not ? In other words, is there a case where decision trees can have an accuracy score strictly lower than 100% for in-sample evaluation ?

Thanks a lot.

$\endgroup$

1 Answer 1

0
$\begingroup$

There is no theoretical reason, only a practical one. Any method can achieve an in-sample accuracy of 100%. It's easy, just make it overfit. Of course, the out-sample accuracy may crash.

This is why the in-sample accuracy score is not used on its own, but with the out-sample score as well, to be able to check that the model is not overfitted.

$\endgroup$
6
  • $\begingroup$ Thanks for your answer. For sure, the out-sample accuracy will not be good, that was not really my point. Actually, the question is rather why for decision trees, the in-sample accuracy is always 100%, without making any change on the data (I did not try to overfit, and it gives 100% accuracy for all data sets) $\endgroup$
    – Othman_J
    Dec 27, 2018 at 8:24
  • $\begingroup$ You can always get up to 100%. For decision trees, it may be that for your dataset, the default hyper parameters make them overfit. $\endgroup$ Dec 27, 2018 at 8:26
  • $\begingroup$ my data set has no particular feature, here I test with the Iris data. You can try with any other data set. the result will be the same : 100% for decision trees, and a score strictly lower than 100% for other methods. There should be a reason I guess? $\endgroup$
    – Othman_J
    Dec 27, 2018 at 8:30
  • $\begingroup$ hyperparameters, that's all. You can make the other datasets overfit as well. Just increase their capacity. $\endgroup$ Dec 27, 2018 at 8:34
  • $\begingroup$ You mean other methods right? Ok, based on the same Iris data set : are you able to provide a code in which Gaussian NB gives 100% for in-sample evaluation OR a code that gives less than 100% for decision trees? $\endgroup$
    – Othman_J
    Dec 27, 2018 at 8:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.