In some boost models there are option to send eval set while fitting the model. for example:

xgb_model = XGBRegressor(n_estimators=1000)
xgb_model.fit(train_X, train_y, early_stopping_rounds=5,
eval_set=[(test_X, test_y)], verbose=False)

In addition, when using Optuna, you should write an objective function that returns a metric score. For example:

 def objective(trial):
        X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
        param = {
            "loss_function": trial.suggest_categorical("loss_function", ["RMSE", "MAE"]),
            "learning_rate": trial.suggest_loguniform("learning_rate", 1e-5, 1e0),
            "l2_leaf_reg": trial.suggest_loguniform("l2_leaf_reg", 1e-2, 1e0),
        reg = CatBoostRegressor(**param, cat_features=categorical_features_indices)
        reg.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=0, early_stopping_rounds=100)
        y_pred = reg.predict(X_test)
        score = r2_score(y_test, y_pred)
        return score

When using both a validation set and a test set, which dataset should be used as eval_set, and which dataset should be used for calculating the metric score in the objective?

  • $\begingroup$ It sounds like you're asking the difference between a test and a validate set. We have lots of questions about this. Here's a search to get you started. stats.stackexchange.com/… $\endgroup$
    – Sycorax
    Jun 23 at 14:46


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.