Unstable variable importance ranking I am new to R and and random regression forest. Right now I am working with a dataset of 60 input variables (dummy variables and continuous variables) and try to find the most important variables, which describe my dependent variable best. Therefore I am using the permutation-based OOB-MSE. 
My problem is now, that each time I run the random forest, the ranking of the variables changes even when I just repeat the command with the same amount of ntree and mtry. First I was thinking that it had something to do with the randomized number of variables used in each tree, so I increased ntree. However, this also didn't help... Has anybody an idea what the problem might be and how I can get stable results for the variable importance measure?
Thanks!
 A: There are a few things that can effect importance.
If the variables in your data set are correlated there can be a lot of instability in the variable importance as the model can use the variables somewhat interchangeably. Ideally it will spread the importance over all of the correlated variables but in practice it may require a lot of trees for this to happen. Reducing mTry or using extra/totally randomized trees is one way to combat this though this may hurt you're prediction accuracy or at least require re tuning...the most accurate model may not be the best for identifying feature importance.
Masking scores and other methods for explicitly dealing with correlated features have also been proposed.
You could also try doing dimensionality reduction before building you're models but this may destroy some of the non linear etc structure in the data depending on how you do it.
There are also biases in CART style feature selection towards "high cardinality" or sparse features. These features tend to be able to produce decreases in impurity that don't generalize well by random chance.
Further I'd only ever expect the importance ranking to be consistent for the top few features which get used across most of the bagged trees. The less useful features will get used less and have a lot more variability.
[2]: Also here: https://eranraviv.com/random-forest-importance-measures-are-not-important/
A: Random Forests are stochastic by nature, so it is normal that the results slightly change from run to run. However, the differences should be very small, especially for measures like Variable Importance (either Gini or mean decrease in accuracy) that are stabilized by being averaged over all trees. 

First I was thinking that it had something to do with the randomized number of variables used in each tree, so I increased ntree

The ntree parameter does not control the size of the subset of variables that are randomly tried at each split. The mtry parameter does.
I suspect that there is an issue with your ntree and mtry parameters. If you try only a few variables at each split, and your ntree is not high enough, some variables won't be given a chance to play a role in each tree. This could lead to some instability in the variable importance measures. This is especially true if you have a low mtry, and a lot of important predictors. But there might be other reasons. 
You should perform a grid search to tune ntree and mtry based on the OOB error rate first. Or, based on a decent number of trees (to ensure convergence), e.g., 500, you can use the tuneRF function to optimize mtry. Then, once you have your best model, you can use its output (e.g., variable importance measures).  
A: I've been dealing with this same issue. Importance rankings change when I randomly stratify on the dependent variable. I looped through 10 to 500 trees random forest models. I get convergence but when I randomly select a training data set the importance rank changed dramatically. So this methodology seems to work well in identifying the top feature selection but it's not stable if you're trying to standardize a methodology for relative importance. Covariance can be measured better in other ways. Random Forests are really good at giving you suggestions. I wouldn't rely solely on a single random forest 
