2
$\begingroup$

I'm new to class imbalance and applying class imbalance technique 'RusBoost' on my dataset. I'm using ebmc package from R. I'm having difficulties to get its arguements values, as per the documentation:

data("iris")
iris <- iris[1:70, ]
iris$Species <- factor(iris$Species, levels = c("setosa", "versicolor"), labels = c("0", "1"))
model2 <- rus(Species ~ ., data = iris, size = 20, alg = "rf", ir = 1, rf.ntree = 100)

Documentation gives the following details about arguments:

  • formula:

    A formula specify predictors and target variable. Target variable should be a factor of 0 and 1. Predictors can be either numerical and categorical.

  • data:

    A data frame used for training the model, i.e. training set.

  • size:

    Ensemble size, i.e. number of weak learners in the ensemble model.

  • alg:

    The learning algorithm used to train weak learners in the ensemble model. cart, c50, rf, nb, and svm are available. Please see Details for more information.

  • ir:

    Imbalance ratio. Specifying how many times the under-sampled majority instances are over minority instances. Interger is not required and so such as ir = 1.5 is allowed.

  • rf.ntree:

    Number of decision trees in each forest of the ensemble model when using rf (Random Forest) as base learner. Integer is required.

This is the link for a documentation for your reference: https://cran.r-project.org/web/packages/ebmc/ebmc.pdf

I couldn't able to figure out size and ir parameter values. How could I? I have a binary classification problem, which predicting: "Yes" , "No". There are 34% "Yes" labeled data and 66% "No" labeled data in my dataset(total 15,000 records).

Thanks in advance.

$\endgroup$
1
$\begingroup$

A 34-66 dataset with $N\approx 15000$ is far from being considered imbalanced. Any reasonable classification algorithm will work fine in this case.

I would suggest focusing on making "probabilistic prediction or classification" where you focus on predicting a reasonable probability of instance $i$ being a member of particular class $A$. In that case, a suitable metric would be the Brier Score. You may be interested in exploring some of the highly voted questions using the tags and to get a better idea.

Addition/Edit following comment: The general advice on choosing hyper-parameters applies here too: Use resampling (either bootstrap or repeated cross-validation) to get estimates of methods $A$ performance. This principal is ubiquitous in all machine learning methodologies. Particular to RusBoost, especially size is in direct analogy with the number of iterations in a most gradient boosting algorithms and cross-validation is the standard way we use to get iter.

We can use the functionality of a package like caret to create relevant folds (see for example the function createFolds) that will be used for training and validating the classifiers performance. Notice that the values of the hyper-parameters will be specific to the problem at hand and they will not be the same for different datasets/applications. (i.e. if we find that for this application 55 C50 learners with ir equals 1 give us the best Brier score we should not go ahead and use these numbers in every application of RusBoost we come across. We should use proper cross-validation and choose new values for our new problem at hand.)

$\endgroup$
  • $\begingroup$ Thanks for providing great suggestions. I will look into it for sure. Especially Brier Score. I've also tried the Random Forest classification, which is giving ROC of 74%, sensitivity of 31% and specificity of 89%. Since I'm working on medical clinical data, both sensitivity and specificity matter. That's why I focused on rusBooting to increase my sensitivity a bit. Even If I choose some another option, I still want to figure out rus function from ebmc package. $\endgroup$ – M_Gandhi Jul 29 '18 at 7:17
  • $\begingroup$ I am glad I could help! I made some substantial edits to my answer so it addresses your comment. As a side-note: I would expect that the choice of the base-learner (C50, CART, RF, etc.) to have a substantial influence in the algorithms performance too. Unless there is a specific learner that should be used I would cross-validate that hyper-parameters too. $\endgroup$ – usεr11852 Jul 29 '18 at 12:37
  • $\begingroup$ I already started to implement repeatedcv and I am glad I heard that from you as I'm on the same path as you suggest. I got your size and iter relationships, will try for sure. I will also definitely try other base learners. Even I am planning to try XGBoosting as well. What're your suggestions on XGBoosting? $\endgroup$ – M_Gandhi Jul 29 '18 at 16:56
  • $\begingroup$ You can use caret to train and obtain repeated-CV results directly. $\endgroup$ – usεr11852 Jul 29 '18 at 18:33
  • $\begingroup$ Yup. I am able to perform cross-validation using caret package repeated-cv. Also, Can you brief on your suggestions for XGBoosting? $\endgroup$ – M_Gandhi Jul 29 '18 at 23:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.