With a colleague, we are working on a dataset containing ~5000 continuous variable for 120 individuals
belonging to 8 classes.
We want to **estimate the relative importance of each variable** to explain the classes.
We have used with some success a random forest approach.
Now, we could like to go deeper by considering the fact that **the 8 classes we fit are unequally distant from each other**.
In fact, in our case **we can *a priori* generate a distance matrix for all possible pairs of classes**.

My (very limited) understanding of random forest is that, for regression problems,
the error $E$ is computed by the mean square difference between the OOB sample and the prediction for the same sample:

$E = n^{-1}\sum\limits_{i=1}^n{{(y_i-\hat{y}_i)}^2}$

Where $y_i$ is the predicted value and $\hat{y}_i$ the real value of an out-of bag-sample $i$.
Ultimately, the calculation of the variable importance depends on how the error is computed (right?).

In our case, I would like to use a modified loss function, for instance:

$E = n^{-1}\sum\limits_{i=1}^n{M_{y_i,\hat{y}_i}}$

Where $M$ is predefined a distance matrix; so $M_{a,b}$ is the distance between class $a$ and $b$.
In this way the misclassification error would be more important if $y_i$ and $\hat{y}_i$ represent distant classes and, ultimately, **the variable importance should be more relevant**.


My questions are:

   1. Does this approach make sense to you, or am I missing something?
   2. Can you think of any study that has used something similar.
   3. We have so far used the `randomForest` package in R. It does not seem possible to use
  it in combination with an a priori distance matrix between classes. Do you know if this is already implemented somewhere?