I'm using the random forest classifier (
scikit-learn on a dataset with around 30 features, 3000 data points and 6 classes. I'm using leave-one-group out as well as leave-one-out cross-validation.
RandomForestClassifier provides directly the importances of the features through the
feature_importances_ attribute. I think the importance scores are calculated by averaging the feature importances of each tree which are retrieved by looking at the impurity.
Currently I'm fitting a random forest on the whole dataset and then I'm looking at the feature importances. Another option would be to retrieve the feature importances on each training set of each split of the cross-validation procedure and then averaging the scores.
Which one is the correct way to go?
Second, how can I calculate if one (or several) features have significant more importance than others (p-value)?