0
$\begingroup$

I have a small dataset with many features, but unfortunately only 19 observations from 2 categories. The idea is that I can determine feature importance in classifying samples in one of 2 categories. For context, I have thousands of features, and am just looking to exclude some of the noisy ones.

I'm aware that there are a few things that I should be careful. Specifically, I should either use crossvalidation or OOBE to get good (accurate) models. Specifically, given that my dataset is so small (9 vs 10 observations), I'd like to avoid 2 or 3-fold CV, and was instead thinking about using OOBE to train my model and use it for feature importance. (I was also looking at this post for more information). For sake of completeness, I also use a regressor.

The problem: unfortunately, when I run my classifier, even using bagging, I still get a stupidly high accuracy of 1.

Question 1: I assumed that by specifying oob_score=True bagging was being performed. Am I misreading this?

Question 2: I also assume that, since bagging is being done, I don't need to split my data into training/testing, hence why I'm checking the scores against the training data. This is something that was also done in the blog post above

Question 3: Do you think I need to try the "column drop" approach given in the blog post I mentioned above?

I appreciate all comments. Below it's the code I'm using

# data and classes
X_train, y_train= data_in, categories
X_train['random'] = np.random.random(19)

# RF classifier
rf_class=sklearn.ensemble.RandomForestClassifier(
    n_estimators=100,
    min_samples_leaf=1,
    random_state=42,
    n_jobs=-1,
    oob_score=True)

# RF regressor
rf_regress=sklearn.ensemble.RandomForestRegressor(
    n_estimators=100,
    min_samples_leaf=1,
    random_state=42,
    n_jobs=-1,
    oob_score=True)

fit_class=rf_class.fit(X_train_in, classes_train_in)
fit_regress=rf_regress.fit(X_train_in, classes_train_in)

scores=[
        fit_class.score(X_train_in, classes_train_in),
        fit_regress.score(X_train_in, classes_train_in)
]
# [1.000000, 0.799444]

oob_scores=[
        fit_class.oob_score_,
        fit_regress.oob_score_
]
# [0.631579, 0.027535]
$\endgroup$
2
$\begingroup$

So you have 19 subjects, 2 categories and many features. You are using a fairly complex model, which is giving you a very high accuracy. So either you have a very clean classification problem or it may be over fitting.

To verify if it I would also look into other scores: specificity and sensitivity. It could be one of the classes is well classified while the other has big errors.

If you have some specialist knowledge on the subject and since you have a small dataset I would do some visualization (e.g. side by side boxplots for class 1 and for class 2 of each feature), to make sure you are not feeding the algorithm with irrelevant data.

$\endgroup$
5
  • 4
    $\begingroup$ The observation that there are only 19 observations is the source of the problem. It should be simple to overfit 19 observations with a complex model. The only solution is to collect much more data. $\endgroup$ – Sycorax Dec 3 '19 at 14:52
  • 1
    $\begingroup$ @SycoraxsaysReinstateMonica I completely agree with you, this has been discussed in this context between myself and others. But if this is not possible, how would you best use this to exclude some features? Can I trust that my excluding those showing null importance (there are many!) can in fact be excluded, or does the number of observations not allow me to say so? $\endgroup$ – Sos Dec 3 '19 at 15:03
  • $\begingroup$ One more point: I noticed now that I should be using oob_score rather than score (see here), which may explain the absurdly high accuracy. When I use oob_score instead I get instead 0.631579 for the classifier for instance $\endgroup$ – Sos Dec 3 '19 at 15:06
  • 2
    $\begingroup$ @Sosi OOB scores are just measured on the left-out observations from each bagging round. Approximately one third of your data is left out at each round, so that's 6 observations per round. Do you think that's enough data to form a sufficiently precise measurement of error for your purposes? You're free to do whatever you want, but if collecting more data is impossible, using this model to do anything of consequence is very risky. $\endgroup$ – Sycorax Dec 3 '19 at 15:12
  • $\begingroup$ @SycoraxsaysReinstateMonica I completely understand and appreciate your comment. Do you have any suggestions for alternative models? $\endgroup$ – Sos Dec 3 '19 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.