0
$\begingroup$

I'm trying to tune hyperparameters for XGBoost using RandomizedSearchCV. I have five outputs and I guess that might be the reason that I can't run RandomizedSearchCV or GridSearchCV to tune hyperparameters. Running this code:

params = { 'max_depth': [3, 5, 6, 10, 15, 20],
           'learning_rate': [0.01, 0.1, 0.2, 0.3],
           'subsample': np.arange(0.5, 1.0, 0.1),
           'colsample_bytree': np.arange(0.4, 1.0, 0.1),
           'colsample_bylevel': np.arange(0.4, 1.0, 0.1),
           'n_estimators': [100, 500, 1000]}

xgbr = xgb.XGBRegressor(seed = 20)

clf = RandomizedSearchCV(estimator=xgbr,
                         param_distributions=params,
                         scoring='neg_mean_squared_error',
                         n_iter=25,
                         verbose=1)

clf.fit(X_train, Y_train)
print("Best parameters:", clf.best_params_)
print("Lowest RMSE: ", (-clf.best_score_)**(1/2.0))

I get this error:

UserWarning: One or more of the test scores are non-finite: [nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan
 nan nan nan nan nan nan nan]
  warnings.warn(

Is having multiple outputs in Y_train the cause of this issue?

Improve this question
$\endgroup$
3
  • $\begingroup$ sklearn's mse handles multioutput, by default just averaging the mse's of each output. Whenever scoring fails in a hyperparameter search, set error_score="raise" so you can see the traceback of the error. $\endgroup$
    – Ben Reiniger
    Commented Nov 14, 2022 at 16:02
  • $\begingroup$ @BenReiniger yes, I have already used mse to evaluate my model with no issues. The problem is no tuning method works for multiple outputs. I'm not sure if I have missed something here. $\endgroup$
    – Hanna
    Commented Nov 14, 2022 at 16:12
  • $\begingroup$ The docs for RandomizedSearchCV.fit say that y can have shape (n_samples, n_output), so it should work for multioutput. Try getting the full error traceback as mentioned before, and/or provide a minimal example (dataset) that produces the error. $\endgroup$
    – Ben Reiniger
    Commented Nov 14, 2022 at 16:15

1 Answer 1

Reset to default
0
$\begingroup$

I guess it might be an incompatibility between the parameters in params.

The output you are getting is caused by a regressor that is generating answers that are not a number, ex: 1/eps where eps can be a very small number.

You have to check fewer params first to see where the problem lies.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.