0
$\begingroup$

What happens if a random forest "max bins" hyperparameter is set too high?

When training a sparkml random forest with maxBins set roughly equal to the max number of distinct categorical values for any given feature I see OK performance metrics. But when I set it closer to 2x or 3x the number of distinct categorical values, performance is terrible (eg. accuracy (in the case of a binary classifier) being no better than just the actual distribution of responses in the dataset) and the feature importances being all zeros (as opposed to when using the lower initial maxBins value where it at does show something for the importances).

I would not think that there would be such a huge difference just from a change in max bins like this (esp. the difference in seeing something vs absolutely nothing / all zeros for the feature importances).

What could be happening under the hood of the algo that causes such different outcomes when this parameter is changed like this?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

It is a decision tree parameter:

maxBins: Number of bins used when discretizing continuous features.

  • Increasing maxBins allows the algorithm to consider more split candidates and make fine-grained split decisions. However, it also increases computation and communication.
  • Note that the maxBins parameter must be at least the maximum number of categories $M$ for any categorical feature.

If you have a categorical variable with $K$ categories, then

  • If $K > M$, what you should do for this to make sense, is to cluster similar categories together. Likely, Spark's implementation does deterministic splits. Categorical variable codes categories as arbitrary numbers, so packing together categories 1 and 2 only because the numbers are close, does not have to mean that such grouping makes any sense at all. If you would like to reduce the dimensionality, you would need to do some kind of clustering to pack similar categories together.
  • If $K = M$ it would be an identify function, you are not doing any binning since each bin has only one value.
  • If $K < M$ you would be packing the same values into different bins. From a feature engineering point of view, it doesn't make sense. Likely they are recommending this because of some implementations details of how they do the distributed training.
Improve this answer
$\endgroup$
8
  • $\begingroup$ I had seen those docs and read the categorical bullet point "must be at least the maximum number of categories" and assumed that there would not be such dramatic difference when choosing a value above the "maximum number of categories". Could you explain more what you mean when you say "They cannot be used, because this would mean putting the same values into different bins, this would lead to confusing results at best."? What would be getting "confused" here? (From your description, it seems like the only good answer would be having the exact same number of bins and max categories). $\endgroup$
    – lampShadesDrifter
    Commented Jun 15, 2021 at 7:30
  • $\begingroup$ @lampShadesDrifter if you have the same values in different bins, then if a decision tree does a split where such bins land on different branches, it would need to make different predictions for the same values. Not sure why they used the "at least" term, it might be about some implementation detail, e.g. spreading values between different workers when doing distributed training. $\endgroup$
    – Tim
    Commented Jun 15, 2021 at 7:46
  • $\begingroup$ As a sidenote, unless you really have to, there are manny better implementations then Spark's for random forest github.com/szilard/benchm-ml $\endgroup$
    – Tim
    Commented Jun 15, 2021 at 7:49
  • 1
    $\begingroup$ @lampShadesDrifter well... first of all, you don't really need to bin the categorical features. If you bin them into the same number of bins as categories, you are not doing anything, it would be an identity function. It might be the case that for some implementations reasons things work faster, or scale better for Spark if you make it higher, but it has nothing to do with the algorithm itself, but the implementation. Maybe you'd get better answer on some kind of Spark support site or Spark users forum. $\endgroup$
    – Tim
    Commented Jun 15, 2021 at 7:53
  • 1
    $\begingroup$ @lampShadesDrifter it's not a name, I meant that they don't use something like clustering for that. Likely they just take all the values and cut them using something like quantiles into bins according to the numerical values. For categorical features numerical values for the codes have no mathematical meaning, so this doesn't make sense. $\endgroup$
    – Tim
    Commented Jun 15, 2021 at 10:40
1
$\begingroup$

"Every worker has to compute summary statistics for every feature and every possible split point, and those statistics will be aggregated across the workers. MLlib requires maxBins to be large enough to handle the discretization of the categorical columns. The default value for maxBins is 32, and we had a categorical column with 36 distinct values, which is why we got the error earlier. While we could increase maxBins to 64 to more accurately represent our continuous features, that would double the number of possible splits for continuous variables, greatly increasing our computation time. Let’s instead set maxBins to be 40 and retrain the pipeline."

I am quoting from the Learning Spark 2nd edition by Jules S. Damji, Brooke Wenig, Tathagata Das, and Denny Lee on Page 311, Hyperparameter Tuning.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.