With a colleague, we are working on a dataset containing ~5000 continuous variable for 120 individuals belonging to 8 classes. We want to estimate the relative importance of each variable to explain the classes. We have used with some success a random forest approach. Now, we could like to go deeper by considering the fact that the 8 classes we fit are unequally distant from each other. In fact, in our case we can a priori generate a distance matrix for all possible pairs of classes.

My (very limited) understanding of random forest is that, for regression problems, the error $E$ is computed by the mean square difference between the OOB sample and the prediction for the same sample:

$E = n^{-1}\sum\limits_{i=1}^n{{(y_i-\hat{y}_i)}^2}$

Where $y_i$ is the predicted value and $\hat{y}_i$ the real value of an out-of bag-sample $i$. Ultimately, the calculation of the variable importance depends on how the error is computed (right?).

In our case, I would like to use a modified loss function, for instance:

$E = n^{-1}\sum\limits_{i=1}^n{M_{y_i,\hat{y}_i}}$

Where $M$ is predefined a distance matrix; so $M_{a,b}$ is the distance between class $a$ and $b$. In this way the misclassification error would be more important if $y_i$ and $\hat{y}_i$ represent distant classes and, ultimately, the variable importance should be more relevant.

My questions are:

1. Does this approach make sense to you, or am I missing something?
2. Can you think of any study that has used something similar.
3. We have so far used the randomForest package in R. It does not seem possible to use it in combination with an a priori distance matrix between classes. Do you know if this is already implemented somewhere?