Loss Matrix Equivalent with Neural Networks and random Forest

Question

I'm doing classification (0,1) on a dataset for which different types of errors should be weighted differently. IE, false positives would be weighted 10 x more than false negatives.

In decision trees, specifically the rpart implementation of CART in R, I can simply pass the function a loss matrix.

[0,1]
[10,0]

which would result in weighting errors differently and build trees based on that.

for random Forest, it SHOULD be possible to do exactly the same, since it's still based on the same algorithm. With neural networks, I would think it'd still be possible, but I would have to modify the error/loss function in such a way to account for the difference in errors.

I'm hoping someone will lead me in the right direction on how to go about this.

Thanks.

Answer 1

For neural networks, my initial gut feeling was that you could simply modify the error or discrepancy function to include a class-specific penalty. For example, suppose you're using the typical minimize-the-sum-squared-error approach, you normally minimize $\sum_i(y_i - o_i)^2$, where $o$ is the network's output and $y$ is the "true" label for example $i$. You could simply scale that by a constant that depends on the true and predicted class. Kukar and Kononenko (1998) looked at a few other approaches and found that this one typically works best.

Cost-sensitive random forests shouldn't be a problem either; they were (briefly) discussed in this thread.

There are about a zillion random forest and neural network implementations floating around though, so it's hard to know if these options have been added to your software package of choice.