With a colleague, we are working on a dataset containing ~5000 continuous variables for 120 individuals belonging to 8 classes.

We want to estimate the relative importance of each variable to explain the classes. We have used a random forest approach with some success. Now, we could like to go deeper by considering the fact that the 8 classes we fit are unequally distant from each other. In fact, in our case we can a priori generate a distance matrix for all possible pairs of classes.

My (very limited) understanding of random forest is that, for regression problems, the error $E$ is computed by the mean square difference between the OOB sample and the prediction for the same sample:

$E = n^{-1}\sum\limits_{i=1}^n{{(y_i-\hat{y}_i)}^2}$

Where $y_i$ is the predicted value and $\hat{y}_i$ the real value of an out-of bag-sample $i$.

Ultimately, the calculation of the variable importance depends on how the error is computed (right?).

In our case, I would like to use a modified loss function, for instance:

$E = n^{-1}\sum\limits_{i=1}^n{M_{y_i,\hat{y}_i}}$

Where $M$ is predefined a distance matrix; so $M_{a,b}$ is the distance between class $a$ and $b$. In this way the misclassification error would be more important if $y_i$ and $\hat{y}_i$ represent distant classes and, ultimately, the variable importance should be more relevant.

My questions are:

1. Does this approach make sense to you, or am I missing something?
2. Can you think of any study that has used something similar.
3. We have so far used the randomForest package in R. It does not seem possible to use it in combination with an a priori distance matrix between classes. Do you know if this is already implemented somewhere?

EDIT

After some research, it appears that my question is about using a cost-sensitive version of random forest. In this respect, it is very similar to this question. I specificity want to use a cost matrix rather than a cost vector though. It there any ontological reason why it is not possible or is it simply not implemented?