4
$\begingroup$

I'm a PhD student in Information Retrieval with some limited experience in ML. We've been working on a binary classification task with weka (I'm using weka programmatically via Java), specifically with Random Forest.

Our results are coming out a little weird because we have an unbalanced dataset (85/15ish). We're getting very high % correct, but precision and recall are very low for our target class (the 15% one).

My new understanding is that % correct is really not the right metric to be looking at. The professor I work with said (and I quote): "You are measuring accuracy with "percent correct". This is so rarely done in machine learning papers these days that I just blew right past it." He also referenced a paper explaining why I shouldn't use % correct as Accuracy [1].

In our case, we are interested in precision and recall to some extent, but the professor I'm working with (he's an ML expert) explained that we can and should use AUC-ROC to compare the runs because that's not sensitive to data balance. After he explained this in depth, I got it and understood. And, I was able to get the AUC data out of the Weka results, which are decent though not spectacular (in the 0.75 neighborhood).

I'm used to IR systems in which you can tune for various metrics, e.g. Precision, F values, MAP, etc. However, as far as I can tell, Weka always trains its classifier models to optimize for % correct. So even though I am interested in another metric, e.g., Precision or F1, I can't for the life of me figure out how to encourage Weka to train its model to focus on optimizing for anything other than % correct (say, F1).

I've combed the weka docs and Googled the heck out of it (incl. site search here on CrossValidated) but couldn't find anything to.

Is that possible? I would really appreciate any insight into whether that's even a possibility at all, is it just not implemented in Weka, or if there's some reason why it shouldn't be done. Or, if there's something I'm missing because I'm calling weka from Java rather than using the GUI.

--

[1] Provost, F. J., Fawcett, T., & Kohavi, R. (1998, July). The case against accuracy estimation for comparing induction algorithms. In ICML (Vol. 98, pp. 445-453). http://eecs.wsu.edu/~holder/courses/cse6363/spr04/pubs/Provost98.pdf

$\endgroup$
1
$\begingroup$

My cursory search did not find this option either. As you describe the problem, you want to use:

  1. An imbalanced dataset (85:15).
  2. Random Forest.
  3. ROC and AUC-based loss definitions.
  4. Weka.

Let's try to relax one condition at a time.

Here are some possible alternatives:

  1. Intentionally skew the data: take all the instances from the 15% label and sample a similar number from the other label. Say you have 850 yellow instances and 150 blue, take all the blue instances and sample 150 yellow ones. Then train a random forest using Weka. You can use bootstrap resampling if you want to diversify the data.
  2. Use a cost-sensitive classifier, and mark the cost of false negatives higher. cost-sensitive classification in Weka
  3. Use a different loss function. Like you, I could not find how to do this for the current framework/algorithm combination.
  4. Use a different algorithm. SGD in Weka can use different loss functions.
  5. Use a different ML framework. scikit-learn seems more flexible, but I am unsure whether its implementation of random forest allows for ROC curve-based loss.
$\endgroup$
  • $\begingroup$ Thanks a lot, Yuval! Indeed these are the same conclusions I reached. "Loss function" was the term I was, ahem, at loss for. Also, another colleague suggested this solution which I'm posting here for others' benefit: Check out stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#balance (the balancing error section). You might be able to weigh your instances (your classes) appropriately, so that you can artificially balance your classes. And this is the way to add weights to a dataset in Weka: weka.wikispaces.com/Add+weights+to+dataset $\endgroup$ – shiri Nov 6 '14 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.