I'm doing classification (0,1) on a dataset for which different types of errors should be weighted differently. IE, false positives would be weighted 10 x more than false negatives.

In decision trees, specifically the rpart implementation of CART in R, I can simply pass the function a loss matrix.


which would result in weighting errors differently and build trees based on that.

for random Forest, it SHOULD be possible to do exactly the same, since it's still based on the same algorithm. With neural networks, I would think it'd still be possible, but I would have to modify the error/loss function in such a way to account for the difference in errors.

I'm hoping someone will lead me in the right direction on how to go about this.



1 Answer 1


For neural networks, my initial gut feeling was that you could simply modify the error or discrepancy function to include a class-specific penalty. For example, suppose you're using the typical minimize-the-sum-squared-error approach, you normally minimize $\sum_i(y_i - o_i)^2$, where $o$ is the network's output and $y$ is the "true" label for example $i$. You could simply scale that by a constant that depends on the true and predicted class. Kukar and Kononenko (1998) looked at a few other approaches and found that this one typically works best.

Cost-sensitive random forests shouldn't be a problem either; they were (briefly) discussed in this thread.

There are about a zillion random forest and neural network implementations floating around though, so it's hard to know if these options have been added to your software package of choice.

  • $\begingroup$ It looks by the thread linked that randomForest() in R doesn't have the necessary means implemented. He gives a gradient boosting method as an alternative. $\endgroup$
    – Faydey
    Commented Apr 27, 2013 at 7:49
  • $\begingroup$ He gives a gradient boosting method as an alternative. I guess I'll try downloading the source for randomForest and see what I'd have to change to give it cost sensitivity. Looking at the documentation on the neuralnet package, I believe It allowed defining a custom loss function. If that's the case, and there's some example I can work from, it shouldn't be too bad to get this done. also, limiting responses to 5 minutes of editing is silly when a carriage return sends the comment in. $\endgroup$
    – Faydey
    Commented Apr 27, 2013 at 7:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.