Random Forest Regressor gives negative test score in GridSearchCV

Question

I built a random forest regressor and used gridserachCV to tune hyperparameters.

    from sklearn.model_selection import GridSearchCV
parameters = {
     'n_estimators':(1, 10, 30, 100),
     'max_depth':(4,5,6,8,10),
     'min_samples_split': (2, 4, 8),
     'min_samples_leaf': (4,8,12)
}
model = GridSearchCV(RandomForestRegressor(random_state = seed),parameters,cv=5,return_train_score=True)
model.fit(X_perf, Y)
model.best_score_, model.best_params_ 

The issue is that my response Y has multiple columns. It has dimension of 16 entries X 10 columns, meaning instead of predicting one output I'm actually predicting 10 at the same time but the data size is only 16. The predictor matrix X_perf is 16 entries X 66 columns. 50 out of 66 are dummy columns created from my original data, which had some unordered categorical predictors.
Now the best_score_ from GridSearchCV is negative, which means the model is definitely overfitting as these "mean test score" are horrible. My question is:

1. I'm trying to predict a bunch of Y at the same time. Is Random Forest Regressor the wrong model? Which model can achieve this?
2. Given I have a very limited data size but a lot of predictors, is there a way to avoid overfitting?

Thank you!

Answer 1

To answer your question, I consulted the following sources:

sklearn.model_selection.GridSearchCV

How does GridSearchCV compute training scores?

sklearn.model_selection.train_test_split

Firstly, it's important to check how you performed the Hold-Out (dividing your data into X_train, X_test, Y_train, Y_test). You can do this by printing and applying .shape.

print('Train:', X_train.shape)
print('Train:', y_train.shape)

After this step, you may consider the possibility that you're incorrectly using train_test_split, here's a brief explanation:

Training: Data you want to learn from, without the target. Testing: Target.

When splitting 80-20, I end up with dividing the training set in this proportion to have unseen data, in case the order is wrong, it can lead to inconsistency. The correct way to split is this and the order:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

...after evaluating the split, you might find your answers. Happy studying :)