Classification in R - seperate category for uncertain classifications

Question

I am constructing classification trees for the first time, so I'm quite new to this use of R.

I have observations of behaviours and incoming data that has to be classified as one of these behaviours. Using the rpart package, I was able to construct a tree:

Tree <- rpart(formula = behaviour ~., data=Tree.Data, method="class")

However, the behaviours I observed do not include all possible behaviours. For example I observed eating and sleeping, but not shaking. Now if my incoming data is actually shaking, I would prefer it to be classified as "unknown" rather than eating (as the best guess of the tree). Additionally, if the tree is not that certain of a prediction (less than 70% sure?), I would like it to be in this "unknown" class too.

I found that I can obtain the probabilities of each incoming datapoint being a certain behaviour:

pred <- predict(Tree, data = incoming.data[,c(behaviour)], type = c("prob"))

However, I do not know how to proceed...

Is there any way within the rpart package to do this? I also looked into Random Forest and thought I might be able to do something with the votes, but I got stuck on this too.

To clarify: I do not want to be able to classify unknown behaviours, I just want to put them on one big "unknown" pile.

Answer 1

You won't be able to get a classification of "Unknown" unless you trained your tree on Unknown behaviors.

And even if you did include Shaking and labeled it "Unknown" in your training sample, your tree would only be able to recognize Shaking - and classify it as "Unknown" - but not some other kind of not-yet-seen behavior, e.g., Swinging From A Chandelier While Reciting Shakespeare.

Donald Rumsfeld's quote about the difference between known unknowns and unknown unknowns is relevant.

What you can do is to look at the classification probabilities, as you already do. You can post-process the probabilities and label anything that does not have a, say, 70% probability for any one class as "Unknown".

If you get more data, you can even try to spot patterns in such low-probability classifications. Maybe if your tree outputs about 40% probability of Eating and 40% of Sleeping, then this is really Shaking, but if the probabilities are 60% and 15%, then we have a case of Swinging From A Chandelier While Reciting Shakespeare. Then again, if you have enough data to run this kind of analysis, you might just as well use the full labeled data to train your tree in the first place.

And Random Forests are a good idea in almost any case, too. Similar arguments apply to them, too.