0
$\begingroup$

Let's say we want to conclude that attributes A, B, C and D are the most relevant attributes to maximize the precision of predicting Y, and then rank those attributes based on their importance/relevance.

Now let's say SVM and Random Forest both seem to be good fits to model the data and Random Forest provides better performance (higher precision). Does that mean Random Forest would also be a better choice to rank the attributes according to their relevance to D?

In a more general sense, can we say the best algorithm for maximizing precision of predicting Y is also the best algorithm to rank the relevance of attributes to Y?

$\endgroup$
  • 2
    $\begingroup$ All variable importance measures tell you only how a specific algorithm used your data. When you report variable importance, you should keep this in mind: "This is a summary of how the algorithm I chose used my training data". There is no concept of variable importance that is algorithm independent that is being measured. Consequently, the variable importance measures you get from different algorithms are not comparable to eachother. $\endgroup$ – Matthew Drury Dec 13 '16 at 19:13
  • $\begingroup$ @MatthewDrury then if the goal is to rank attribute relevance, could somehow averaging the weight of multiple algorithms be a good approach? Like if both SVM and RF are saying A is the most relevant, could we say A is potentially the most relevant? How do you compare this approach to a simple correlation test between A and Y? $\endgroup$ – Aliweb Dec 13 '16 at 19:17
  • 1
    $\begingroup$ As the poster of this question: stats.stackexchange.com/questions/202277/… I'm not sure I'm the best person to help you. I've stopped believing that it's even possible to rank the influence of predictors in a model independent way. $\endgroup$ – Matthew Drury Dec 13 '16 at 19:52
0
$\begingroup$

It is generally recommended to use more than one method for feature selection. For instance, taking a combination of outputs from Recursive feature elimination and Random forests can be effective to select final set of features. There is no general technique for feature selection. A list of possible feature selection techniques along with examples can be found here

$\endgroup$
  • $\begingroup$ But the Q is about measuring importance, not selection $\endgroup$ – kjetil b halvorsen Mar 15 at 22:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.