Interpretation of Random Forests [duplicate]

Question

I have a question regarding interpretation of results of a random forest that I created. First, some background regarding the data:

I have a dataset that consists of 100 true instances and 1000 instances of a negative set. I have 20,000 features, thus giving me a matrix of 1100 x 20,000. (These features are sparsely distributed throughout the matrix. Of the 22,000,000 values of the matrix, 21 million are 0's and only about 1 million have values greater than 0).From this sparse data I created a random forest of 500 trees (using the default settings in package RandomForest in R).

After this training, I tested the random forest on a testing set consisting of another gold standard and negative standard that the RF was not trained on to see how well it performed. The final output for this testing set for each 1100 instances is a probability from zero to 1, of it being in the gold standard. The results were excellent,with ROC curves showing that the Random Forest classifier had a true positive rate of 60% and a false positive rate of less than 5%, which is excellent in this context and outperforms other methods I used previously for this problem.

The problem that I have is I need to justify in this context why each prediction is being made, which I am not sure how to do. For example, if instance 212 is predicted to be part of the gold standard with a probability of 63%, how do I justify why, as concretely as possible, this is being predicted?

By way of comparison, for Naive Bayes Classifiers, this is easily done, since for "instance 212" each feature is associated with an odds ratio (or posterior probability) of that instance being in the gold standard, which makes interpretation easy. Is there an analog of this for Random Forests? I know that there is variable importance, but that does not directly bear on any individual instance.

EDIT:My question does overlap somewhat with How to make Random Forests more interpretable?. However, it sounds like some people there are just saying the answer is no, while others are saying there might be a way to interpret the probability distributions for each tree in a way that is logical, but I am not sure how that would be done.

Answer 1

Let's say that each sample $i$ is represented by values $x_{i,1},x_{i,2},\ldots,x_{i,n}$ where $n$ is total number of features. In your case $n=20,000$.

Let's say that $X_1,X_2,\ldots,X_n$ are random variables that take values in feature 1, 2, ..., $n$.

Also let's say that each instance/sample $i$ has a classification label that corresponds to it $y_i$. Similarly, $Y$ is a random variable that takes values in the set of labels.

Suppose that your testing instance is $212$ (the number you have chosen). This means that you have values $x_{212,1},x_{212,2},\ldots,x_{212,n}$, each describing a feature extracted from instance $212$.

Both Naive Bayes and Random Forests try to answer this question: what is the label of $y_{212}$?

To answer that question, both Naive Bayes and Random Forests try to find $\hat y_{212}$ which maximizes the conditional joint density function:

\begin{equation} \hat y_{212} = \underset{y \in \mathcal{Y}}{\text{arg max }}f_{X_1, X_2, \ldots, X_n| Y}(x_{212,1}, x_{212,2}, \ldots, x_{212,n}| y) \end{equation}

That's ugly to look at. So let's rewrite that but with some notational abuse: \begin{equation} \hat y_{212} = \underset{y \in \mathcal{Y}}{\text{arg max }}f(x_{212,1},x_{212,2}, \ldots, x_{212,n}| y) \end{equation}

Hell yeah dude. That's the dope.

Here is where Naive Bayes and Random Forests differ:

• Naive Bayes shamelessly assumes that: \begin{equation} f(x_{212,1},x_{212,2}, \ldots, x_{212,n}| y) \approx \prod_{f=1}^n\hat f(x_{212,f}|y) \end{equation} Obviously that assumption is usually incorrect as stuff appear to be more dependent on one another in real life. Anyway NB plugs that shameless approximation into the maximization problem and tries to solve it: \begin{equation} \underset{\text{NB}}{\hat y_{212}} = \underset{y \in \mathcal{Y}}{\text{arg max }}\prod_{f=1}^n \hat f(x_{212,f}|y) \end{equation} So basically NB only tries to estimate $f(x_{212,f})$ by $\hat f(x_{212,f})$ for each feature in isolation, assuming that they are totally independent.

• Random Forests -on the other hand- do not perform that shameless simplistic assumption of NB. Instead, it dares to estimate the real thing. But since estimating the real thing is a complicated task, it uses a randomized algorithm to explore the space of solutions in a randomized manner so that it becomes computationally feasible. \begin{equation} f(x_{212,1},x_{212,2}, \ldots, x_{212,n}| y) \approx \hat f(x_{212,1},x_{212,2}, \ldots, x_{212,n}| y) \end{equation}

So how does Random Forests estimate $\hat f(x_{212,1},x_{212,2}, \ldots, x_{212,n}| y)$? The answer is: by creating many (in your case $500$) randomized histograms, and then using the average histogram to be the $\hat f(x_{212,1},x_{212,2}, \ldots, x_{212,n}| y)$.

In other words, those randomized tree splits, are essentially creating randomized partitions in the space where $x_{212}$ is sampled from. Such randomized partitioning eventually implies the creation of some randomized bin in a randomized histogram.