# Feature selection: permutation test Vs deleting a variable

In feature selection for predictive models, it is usually applied a permutation test. In this test, all the values of one variable are randomly permuted and the prediction accuracy is extracted for each permutation. For example, if we have 3 variables (features), the ACC with all variables is 0.9. And, in the permutation test, we get ACC of 0.2, 0.1 and 0.9, respectively, when the first, second and third variables are permuted. Thus, the third variable can be dropped because it does not help for the prediction.

However, if we delete the variable or change all the values to zero, instead of randomly permuting the values, do we get the same result as for the permutation test?

Permuting a single variable does not chance the characteristics of your dataset. If your dataset has $n$ records and $m$ features it will still have $n$ records and $m$ features if you permute one of them. If you delete one feature or set it to 0, the resulting dataset will have $m-1$ features. This is a subtle point: the accuracy on a dataset with $m$ features and the one on a dataset with $m-1$ are not directly comparable.
For example, lets say we have 2 features $F_1$ binary and very predictive and $F_2$ with lots of categories and not predictive at all. RF is known to be biased towards $F_2$ because of the many categories.
• If you permute it, the characteristics of your dataset do not change and the difference in RF accuracy is just due to the predictiveness of $F_2$;