Get the number of weak learner - ebmc package of R - implementing class imbalance RusBoost on my dataset

Question

I'm new to class imbalance and applying class imbalance technique 'RusBoost' on my dataset. I'm using ebmc package from R. I'm having difficulties to get its arguements values, as per the documentation:

data("iris")
iris <- iris[1:70, ]
iris$Species <- factor(iris$Species, levels = c("setosa", "versicolor"), labels = c("0", "1"))
model2 <- rus(Species ~ ., data = iris, size = 20, alg = "rf", ir = 1, rf.ntree = 100)

Documentation gives the following details about arguments:

• formula:

  A formula specify predictors and target variable. Target variable should be a factor of 0 and 1. Predictors can be either numerical and categorical.

• data:

  A data frame used for training the model, i.e. training set.

• size:

  Ensemble size, i.e. number of weak learners in the ensemble model.

• alg:

  The learning algorithm used to train weak learners in the ensemble model. cart, c50, rf, nb, and svm are available. Please see Details for more information.

• ir:

  Imbalance ratio. Specifying how many times the under-sampled majority instances are over minority instances. Interger is not required and so such as ir = 1.5 is allowed.

• rf.ntree:

  Number of decision trees in each forest of the ensemble model when using rf (Random Forest) as base learner. Integer is required.

This is the link for a documentation for your reference: https://cran.r-project.org/web/packages/ebmc/ebmc.pdf

I couldn't able to figure out size and ir parameter values. How could I? I have a binary classification problem, which predicting: "Yes" , "No". There are 34% "Yes" labeled data and 66% "No" labeled data in my dataset(total 15,000 records).

Thanks in advance.

Answer 1

A 34-66 dataset with $N\approx 15000$ is far from being considered imbalanced. Any reasonable classification algorithm will work fine in this case.

I would suggest focusing on making "probabilistic prediction or classification" where you focus on predicting a reasonable probability of instance $i$ being a member of particular class $A$. In that case, a suitable metric would be the Brier Score. You may be interested in exploring some of the highly voted questions using the tags scoring-rules and unbalanced-classes to get a better idea.

Addition/Edit following comment: The general advice on choosing hyper-parameters applies here too: Use resampling (either bootstrap or repeated cross-validation) to get estimates of methods $A$ performance. This principal is ubiquitous in all machine learning methodologies. Particular to RusBoost, especially size is in direct analogy with the number of iterations in a most gradient boosting algorithms and cross-validation is the standard way we use to get iter.

We can use the functionality of a package like caret to create relevant folds (see for example the function createFolds) that will be used for training and validating the classifiers performance. Notice that the values of the hyper-parameters will be specific to the problem at hand and they will not be the same for different datasets/applications. (i.e. if we find that for this application 55 C50 learners with ir equals 1 give us the best Brier score we should not go ahead and use these numbers in every application of RusBoost we come across. We should use proper cross-validation and choose new values for our new problem at hand.)