Variable Importance for Caret Random Forest Regression I have trouble understanding the exact meaning of the feature importance scores in caret for RF regression. As you know there are many potential importance measures for RF. However, there is no clear indication which one is used.
Here is a toy example:
data(iris)

y_train = iris['Sepal.Length']
X_train = iris[2:4]

mdl_rf_inner <- caret::train(X_train, y_train$Sepal.Length, method = "rf",
                             preProcess = c("center", "scale"),
                             ntrees = 1000, importance = T)

feat_imp_2 <- caret::varImp(mdl_rf_inner, scale=F)

Resulting in:
rf variable importance

             Overall
Petal.Length   48.51
Sepal.Width    23.67
Petal.Width    17.15

Please keep in mind that I am predicting sepal length, so despite using iris data it is a regression problem. I read the docs and there is no clear indication as to which variable importance is being calculated (Gini-impurity decrease?, mse decrease?, permuation importance?, out of bag?, etc., etc.).
To further complicate things, the train function also has the importance = T argument, which doesn't really seem to serve a clear purpose when using varImp(). Is that correct?
I would greatly appreciate your insights on this.
Best wishes!
 A: This explanation might be what you missed.

If there is no model-specific way to estimate importance (or the
argument useModel = FALSE is used in varImp) the importance of each
predictor is evaluated individually using a “filter” approach.
For classification, ROC curve analysis is conducted on each predictor.
For two class problems, a series of cutoffs is applied to the
predictor data to predict the class. The sensitivity and specificity
are computed for each cutoff and the ROC curve is computed. The
trapezoidal rule is used to compute the area under the ROC curve. This
area is used as the measure of variable importance. For multi-class
outcomes, the problem is decomposed into all pair-wise problems and
the area under the curve is calculated for each class pair (i.e. class
1 vs. class 2, class 2 vs. class 3 etc.). For a specific class, the
maximum area under the curve across the relevant pair-wise AUC’s is
used as the variable importance measure.
For regression, the relationship between each predictor and the
outcome is evaluated. An argument, nonpara, is used to pick the model
fitting technique. When nonpara = FALSE, a linear model is fit and the
absolute value of the t-value for the slope of the predictor is used.
Otherwise, a loess smoother is fit between the outcome and the
predictor. The R2 statistic is calculated for this model against the
intercept only null model. This number is returned as a relative
measure of variable importance.

