7
$\begingroup$

In feature selection for predictive models, it is usually applied a permutation test. In this test, all the values of one variable are randomly permuted and the prediction accuracy is extracted for each permutation. For example, if we have 3 variables (features), the ACC with all variables is 0.9. And, in the permutation test, we get ACC of 0.2, 0.1 and 0.9, respectively, when the first, second and third variables are permuted. Thus, the third variable can be dropped because it does not help for the prediction.

However, if we delete the variable or change all the values to zero, instead of randomly permuting the values, do we get the same result as for the permutation test?

$\endgroup$
4
$\begingroup$

I think the result might be very similar, unless the classification algorithm you are using is biased.

Permuting a single variable does not chance the characteristics of your dataset. If your dataset has $n$ records and $m$ features it will still have $n$ records and $m$ features if you permute one of them. If you delete one feature or set it to 0, the resulting dataset will have $m-1$ features. This is a subtle point: the accuracy on a dataset with $m$ features and the one on a dataset with $m-1$ are not directly comparable.

Random forests (RF) usually uses the permutation approach: in order to compute the importance of a feature, we compare the decrease in accuracy after permutation. I guess that if you just delete that feature you are a bit less confident in comparing the resulting accuracy.

For example, lets say we have 2 features $F_1$ binary and very predictive and $F_2$ with lots of categories and not predictive at all. RF is known to be biased towards $F_2$ because of the many categories.

  • If you permute it, the characteristics of your dataset do not change and the difference in RF accuracy is just due to the predictiveness of $F_2$;
  • If you delete it, the difference in RF accuracy might be higher because it takes into account that you helped the RF in decreasing its bias.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.