Feature importances in random forest I'm using the random forest classifier (RandomForestClassifier) from scikit-learn on a dataset with around 30 features, 3000 data points and 6 classes. I'm using leave-one-group out as well as leave-one-out cross-validation.
RandomForestClassifier provides directly the importances of the features through the feature_importances_ attribute. I think the importance scores are calculated by averaging the feature importances of each tree which are retrieved by looking at the impurity.
Currently I'm fitting a random forest on the whole dataset and then I'm looking at the feature importances. Another option would be to retrieve the feature importances on each training set of each split of the cross-validation procedure and then averaging the scores.
Which one is the correct way to go?
Second, how can I calculate if one (or several) features have significant more importance than others (p-value)?
 A: In Random forest, generally the feature importance is computed based on out-of-bag (OOB) error. 
To compute the feature importance, the random forest model is created and then the OOB error is computed. This is followed by permuting (shuffling) a feature and then again the OOB error is computed. Like wise, all features are permuted one by one. Based on the increase (which is the score) in the OOB error, the feature importance is estimated. Higher this increase, higher the importance. These scores are then divided by the standard deviation of all the increases.
Apart from this, gini impurity measure can also used to estimate feature importance.   
For more details refer this lecture note.

Which one is the correct way to go?

To compute the feature importance, you can use the complete dataset as the importance estimate is computed based on the OOB observations which are actually the left out observations during the bootstrap aggregation process for each tree in a Random Forest.

Second, how can I calculate if one (or several) features have
  significant more importance than others (p-value)?

Consider the feature importance values to compare the relevance of the features. To get p-value, statistical tests such as ANOVA (for parametric) or Kruskal-Wallis test (for non-parametric) could be used.
