I'm fitting and evaluating a CatBoostRegressor
and a XGBRegressor
to the same regression problem. I tried matching their hyperparameters as closely as possible, yet I'm seeing something strange: catboost test error is monotonically decreasing! Why is that? NOTE that I switched off the fancy boosting_type
and it's plain
.
catboost get_all_params:
{'nan_mode': 'Min',
'eval_metric': 'RMSE',
'iterations': 1000,
'sampling_frequency': 'PerTree',
'fold_permutation_block': 0,
'leaf_estimation_method': 'Newton',
'boosting_type': 'Plain',
'feature_border_type': 'GreedyLogSum',
'bayesian_matrix_reg': 0.1000000015,
'l2_leaf_reg': 3,
'random_strength': 1,
'rsm': 1,
'boost_from_average': True,
'model_size_reg': 0.5,
'approx_on_full_history': False,
'subsample': 0.8000000119,
'use_best_model': True,
'random_seed': 0,
'depth': 6,
'has_time': False,
'fold_len_multiplier': 2,
'border_count': 254,
'classes_count': 0,
'sparse_features_conflict_fraction': 0,
'leaf_estimation_backtracking': 'AnyImprovement',
'best_model_min_trees': 1,
'model_shrink_rate': 0,
'loss_function': 'RMSE',
'learning_rate': 0.009999999776,
'score_function': 'Cosine',
'task_type': 'CPU',
'leaf_estimation_iterations': 1,
'bootstrap_type': 'MVS',
'permutation_count': 4}
xgboost get_params:
{'base_score': 0.5,
'booster': 'gbtree',
'colsample_bylevel': 1,
'colsample_bynode': 1,
'colsample_bytree': 1,
'gamma': 0,
'importance_type': 'gain',
'learning_rate': 0.01,
'max_delta_step': 0,
'max_depth': 6,
'min_child_weight': 1,
'missing': None,
'n_estimators': 1000,
'n_jobs': 1,
'nthread': None,
'objective': 'reg:linear',
'random_state': 0,
'reg_alpha': 0,
'reg_lambda': 1,
'scale_pos_weight': 1,
'seed': None,
'silent': False,
'subsample': 1,
'verbosity': 1}
catboost
is still learning (very slowly) at 700 trees. The minimum test MSE is about 1.2 forxgboost
while the last test MSE forcatbost
seems to be about 1.4; see if going out to a few thousand trees finds a minimum closer to 1.2 forcatboost
. $\endgroup$xgboost
achieves a substantially better minima way sooner. $\endgroup$