1
$\begingroup$

I have developed a multi-class Random forest model and it’s working great (well, almost too good). I am getting very high accuracy, sometimes even 1. But I am kind of suspicious about this result. The main reason is that even if I train the model with only 1% data, and test with 99%, I still get ~1 accuracy. According to my understanding, this is very odd. That is why I am trying to figure out what’s going and what can be possible reasons behind such behavior.

My dataset has ~63k rows and ~80 columns. But I am using top 30 columns(after feature selection) to train the model. Also there are 13 different classes(labels).

confusion matrix:

[[ 119    0    0    0    0    0    0    0    0    0    0    0    0]
[   0   93    0    0    0    0    0    0    0    0    0    0    0]
[   0    0  158    0    0    0    0    0    0    0    0    0    0]
[   0    0    0  444    0    0    0    0    0    0    0    0    0]
[   0    0    0    0  301    0    0    0    0    0    0    0    0]
[   0    0    0    0    0 3425    0    0    0    0    0    0    0]
[   0    0    0    0    0    0 6702    0    0    0    0    0    0]
[   0    0    0    0    0    0    0  727    0    0    0    0    0]
[   0    0    0    0    0    0    0    0   96    0    0    0    0]
[   0    0    0    0    0    0    0    0    0  116    0    0    0]
[   0    0    0    0    0    0    0    0    0    0  119    1    0]
[   0    0    0    0    0    0    0    0    0    0    0   97    0]
[   0    0    0    0    0    0    0    0    0    0    0    0  260]]

My code:

df = pd.read_csv("merged_data_set.csv")
df = df[(df[['4', '5', '9']] > 0).all(1)]
df = df.reset_index(drop=True)
print(df.head)
df=df.dropna()
print(df.head)

X = df.drop(df.columns[len(df.columns)-1], 1)
Y = df[df.columns[len(df.columns)-1]]
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=42)
from sklearn import tree
clf=RandomForestClassifier(n_estimators=500)
clf.fit(X_train,y_train)
y_pred=clf.predict(X_test)
print("Accuracy:",metrics.accuracy_score(y_test, y_pred))
print(metrics.confusion_matrix(y_test, y_pred))
$\endgroup$
  • $\begingroup$ How many rows belong to each of the 13 classes? $\endgroup$ – Bryan Krause Jun 29 at 21:56
  • $\begingroup$ data is not evenly distributed, the lowest is ~480 rows, and heights is 33k $\endgroup$ – Masudul Hasan Jun 29 at 21:57
  • $\begingroup$ That could contribute; what does a confusion matrix look like? could it be that your model is doing quite well by just sorting a couple dominant categories and basically ignoring all the others? $\endgroup$ – Bryan Krause Jun 29 at 22:00
  • 2
    $\begingroup$ Update your question with the feature selection code, don’t reply in the comments $\endgroup$ – astel Jun 29 at 23:53
  • 1
    $\begingroup$ But yes you’re probably leaking data when you do feature selection (among other places most likely). Try running the forest on all 80 variables and see what accuracy you get. $\endgroup$ – astel Jun 29 at 23:57
1
$\begingroup$

It is possible to have some data leaking problems, i.e., there are some "cheating" variables in the features. For example, we are prediction if a user will buy some product, but "mistakenly" use number of customer call a person had as a feature.

A related question and answer can be found here. You may quickly fit a decision tree to see if there are very indicative features.

How can I quickly detect cheating variables in large data?

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.