# Random Forest classifier with mean accuracy of 1? Sounds fishy

I have a small dataset with many features, but unfortunately only 19 observations from 2 categories. The idea is that I can determine feature importance in classifying samples in one of 2 categories. For context, I have thousands of features, and am just looking to exclude some of the noisy ones.

I'm aware that there are a few things that I should be careful. Specifically, I should either use crossvalidation or OOBE to get good (accurate) models. Specifically, given that my dataset is so small (9 vs 10 observations), I'd like to avoid 2 or 3-fold CV, and was instead thinking about using OOBE to train my model and use it for feature importance. (I was also looking at this post for more information). For sake of completeness, I also use a regressor.

The problem: unfortunately, when I run my classifier, even using bagging, I still get a stupidly high accuracy of 1.

Question 1: I assumed that by specifying oob_score=True bagging was being performed. Am I misreading this?

Question 2: I also assume that, since bagging is being done, I don't need to split my data into training/testing, hence why I'm checking the scores against the training data. This is something that was also done in the blog post above

Question 3: Do you think I need to try the "column drop" approach given in the blog post I mentioned above?

I appreciate all comments. Below it's the code I'm using

# data and classes
X_train, y_train= data_in, categories
X_train['random'] = np.random.random(19)

# RF classifier
rf_class=sklearn.ensemble.RandomForestClassifier(
n_estimators=100,
min_samples_leaf=1,
random_state=42,
n_jobs=-1,
oob_score=True)

# RF regressor
rf_regress=sklearn.ensemble.RandomForestRegressor(
n_estimators=100,
min_samples_leaf=1,
random_state=42,
n_jobs=-1,
oob_score=True)

fit_class=rf_class.fit(X_train_in, classes_train_in)
fit_regress=rf_regress.fit(X_train_in, classes_train_in)

scores=[
fit_class.score(X_train_in, classes_train_in),
fit_regress.score(X_train_in, classes_train_in)
]
# [1.000000, 0.799444]

oob_scores=[
fit_class.oob_score_,
fit_regress.oob_score_
]
# [0.631579, 0.027535]


• One more point: I noticed now that I should be using oob_score rather than score (see here), which may explain the absurdly high accuracy. When I use oob_score instead I get instead 0.631579 for the classifier for instance – Sos Dec 3 '19 at 15:06