With a colleague, we are working on a dataset containing ~5000 continuous variables for 120 individuals belonging to 8 classes.

We want to **estimate the relative importance of each variable** to explain the classes.
We have used a random forest approach with some success.
Now, we could like to go deeper by considering the fact that **the 8 classes we fit are unequally distant from each other**.
In fact, in our case **we can *a priori* generate a distance matrix for all possible pairs of classes**.

My (very limited) understanding of random forest is that, for regression problems,
the error $E$ is computed by the mean square difference between the OOB sample and the prediction for the same sample:

$E = n^{-1}\sum\limits_{i=1}^n{{(y_i-\hat{y}_i)}^2}$

Where $y_i$ is the predicted value and $\hat{y}_i$ the real value of an out-of bag-sample $i$.

Ultimately, the calculation of the variable importance depends on how the error is computed (right?).

In our case, I would like to use a modified loss function, for instance:

$E = n^{-1}\sum\limits_{i=1}^n{M_{y_i,\hat{y}_i}}$

Where $M$ is predefined a distance matrix; so $M_{a,b}$ is the distance between class $a$ and $b$.
In this way the misclassification error would be more important if $y_i$ and $\hat{y}_i$ represent distant classes and, ultimately, **the variable importance should be more relevant**.


My questions are:

   1. Does this approach make sense to you, or am I missing something?
   2. Can you think of any study that has used something similar.
   3. We have so far used the `randomForest` package in R. It does not seem possible to use it in combination with an a priori distance matrix between classes. Do you know if this is already implemented somewhere?
  
**EDIT**

After some research, it appears that my question is about using a **cost-sensitive** version of random forest. In this respect, it is very similar to [this question][1].
I specificity want to use a **cost matrix** rather than a cost vector though.
It there any ontological reason why it is not possible or is it simply not implemented?

  [1]: http://stats.stackexchange.com/questions/46963/how-to-control-the-cost-of-misclassification-in-random-forests