Random Forest classifier with mean accuracy of 1? Sounds fishy

Question

I have a small dataset with many features, but unfortunately only 19 observations from 2 categories. The idea is that I can determine feature importance in classifying samples in one of 2 categories. For context, I have thousands of features, and am just looking to exclude some of the noisy ones.

I'm aware that there are a few things that I should be careful. Specifically, I should either use crossvalidation or OOBE to get good (accurate) models. Specifically, given that my dataset is so small (9 vs 10 observations), I'd like to avoid 2 or 3-fold CV, and was instead thinking about using OOBE to train my model and use it for feature importance. (I was also looking at this post for more information). For sake of completeness, I also use a regressor.

The problem: unfortunately, when I run my classifier, even using bagging, I still get a stupidly high accuracy of 1.

Question 1: I assumed that by specifying oob_score=True bagging was being performed. Am I misreading this?

Question 2: I also assume that, since bagging is being done, I don't need to split my data into training/testing, hence why I'm checking the scores against the training data. This is something that was also done in the blog post above

Question 3: Do you think I need to try the "column drop" approach given in the blog post I mentioned above?

I appreciate all comments. Below it's the code I'm using

# data and classes
X_train, y_train= data_in, categories
X_train['random'] = np.random.random(19)

# RF classifier
rf_class=sklearn.ensemble.RandomForestClassifier(
    n_estimators=100,
    min_samples_leaf=1,
    random_state=42,
    n_jobs=-1,
    oob_score=True)

# RF regressor
rf_regress=sklearn.ensemble.RandomForestRegressor(
    n_estimators=100,
    min_samples_leaf=1,
    random_state=42,
    n_jobs=-1,
    oob_score=True)

fit_class=rf_class.fit(X_train_in, classes_train_in)
fit_regress=rf_regress.fit(X_train_in, classes_train_in)

scores=[
        fit_class.score(X_train_in, classes_train_in),
        fit_regress.score(X_train_in, classes_train_in)
]
# [1.000000, 0.799444]

oob_scores=[
        fit_class.oob_score_,
        fit_regress.oob_score_
]
# [0.631579, 0.027535]

Answer 1

So you have 19 subjects, 2 categories and many features. You are using a fairly complex model, which is giving you a very high accuracy. So either you have a very clean classification problem or it may be over fitting.

To verify if it I would also look into other scores: specificity and sensitivity. It could be one of the classes is well classified while the other has big errors.

If you have some specialist knowledge on the subject and since you have a small dataset I would do some visualization (e.g. side by side boxplots for class 1 and for class 2 of each feature), to make sure you are not feeding the algorithm with irrelevant data.