Suppose I am doing random forest classification of labels $A$,$B$,$C$,$D$. There is some theoretical ordering to this output such that when $A$ is more likely than $B$, $B$ is also more likely than $C$, etc. Also, if $P(D) > P(C)$, we also have that $P(C) > P(B) > P(A)$. There are other such conditions that need to be met.

The issue is that a real random forest may give something silly that completely violates the above constraints, even if it is able to predict the most likely outcome successfully. For my use case the ordering is important since decisions are made not only on the most likely outcome.

It also seems intuitive that I should be able to improve generalization if I can somehow enforce this prior knowledge into the model.

How do I account for this in a decision forest? Despite this structure to the output I do not think it is possible to construct a real-valued response variable since they are still class labels with no natural real value, even if there is some type of ordering to them.


Here is a possibility: you could add a constraint to the optimization of the purity index (e.g. Gini Index or Entropy) to the individual trees in the forest. So: $$min\,\Sigma{D_i} \; with\;D_i=1-\Sigma^{k}p_{ik}^2$$ $$s.t.\, p_{ik} >= p_{i(k-1)} >= ... >= p_0$$ where $k$ indexes the observation type, $i$ indexes the terminal node and $p_{ik}$ is the proportion of of $k$ on node $i$. That way your forest should yield results consistent with that as well. I guess you could relax the condition by introducing a slack variable $min\,\zeta_i$ with $p_0>\zeta_0 > 0$ $p_0-\zeta_0 <= p_1-\zeta_1$, etc. for the other probs.

But if your data is correct and makes sense and that condition is true for sure your forest will yield results that are consistent with that condition. If you do an unconstrained forest with enough trees and you do not observe your $P(A) < P(B) < ...$ it is quite likely you are mixing non-comparable data sets or that the condition is simply not true.

  • $\begingroup$ The condition is true, irrespective of the data. However the data is mostly noise so it is not the case that I could simply estimate more trees and have the forest learn the condition, since the condition is not reflected unambiguously in the data. So I want to impose this condition 'from above'. $\endgroup$ – Jase Jan 30 '14 at 17:17
  • $\begingroup$ I am not suggesting estimating more trees. I am suggesting putting this condition in the individual tree estimation of your forest. If the condition is true but the data mostly noise, the data is not going to prove it is. You can force your condition into the classification result using the concept above but then I am not sure your data actually shows the condition to be true. $\endgroup$ – Hans Roggeman Jan 30 '14 at 17:23
  • $\begingroup$ My point is that if your data is simply noised data that follows that condition it will show up from the random forest with unconstrained trees. If it does not show up from the unconstrained forest there are probably other factors at play (a non-random unobserved noise that affects the probabilities). $\endgroup$ – Hans Roggeman Jan 30 '14 at 17:29
  • 1
    $\begingroup$ My problem is kind of like modelling hand strength in poker. If I feed it some statistics about pre-flop play and it tells me that QQ is better than AA, I know that something went wrong in training (insufficient data, bad model, etc) and it is good to force this condition on the output. $\endgroup$ – Jase Jan 30 '14 at 17:33
  • 1
    $\begingroup$ That is a great analogy. Imagine data on poker plays that have QQ hands beating AA hand on average. What the data would say is that you have a better chance out bluffing your opponent with a QQ than with any other pair and that effect outweighs the confidence effect instilled by an actual AA hand. However, if you have enough data your random forest is unlikely to say that. And if still says that it is more likely there is another factor (like the chance to cheat whenever I have QQ). $\endgroup$ – Hans Roggeman Jan 30 '14 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.