I have a small dataset with many features, but unfortunately only 19 observations from 2 categories. The idea is that I can determine feature importance in classifying samples in one of 2 categories. For context, I have thousands of features, and am just looking to exclude some of the noisy ones.
I'm aware that there are a few things that I should be careful. Specifically, I should either use crossvalidation or OOBE to get good (accurate) models. Specifically, given that my dataset is so small (9 vs 10 observations), I'd like to avoid 2 or 3-fold CV, and was instead thinking about using OOBE to train my model and use it for feature importance. (I was also looking at this post for more information). For sake of completeness, I also use a regressor.
The problem: unfortunately, when I run my classifier, even using bagging, I still get a stupidly high accuracy of 1.
Question 1: I assumed that by specifying
oob_score=True bagging was being performed. Am I misreading this?
Question 2: I also assume that, since bagging is being done, I don't need to split my data into training/testing, hence why I'm checking the scores against the training data. This is something that was also done in the blog post above
Question 3: Do you think I need to try the "column drop" approach given in the blog post I mentioned above?
I appreciate all comments. Below it's the code I'm using
# data and classes X_train, y_train= data_in, categories X_train['random'] = np.random.random(19) # RF classifier rf_class=sklearn.ensemble.RandomForestClassifier( n_estimators=100, min_samples_leaf=1, random_state=42, n_jobs=-1, oob_score=True) # RF regressor rf_regress=sklearn.ensemble.RandomForestRegressor( n_estimators=100, min_samples_leaf=1, random_state=42, n_jobs=-1, oob_score=True) fit_class=rf_class.fit(X_train_in, classes_train_in) fit_regress=rf_regress.fit(X_train_in, classes_train_in) scores=[ fit_class.score(X_train_in, classes_train_in), fit_regress.score(X_train_in, classes_train_in) ] # [1.000000, 0.799444] oob_scores=[ fit_class.oob_score_, fit_regress.oob_score_ ] # [0.631579, 0.027535]