I would like to perform some sort of binary classification, and my data set consists of 100 examples (for each class), which are vectors with 2500 elements. Ideally, I would like to determine which the most important features are (i.e. best discriminating between those two classes) and reduce the number of features to a more reasonable dimension, e.g. 50.

I decided to start off with a naïve approach, using the Standard score (by only determining the indices of the most prominent features, but without modifying the actual values), which obviously does not take into account the distribution of values per each class. What I basically do is I calculate the score and order in descending order of prominence, and store the indices. Then I just keep the first $n$ (say 50, as suggested above) of them and extract only those features.

Just recently, it was recommended to me to take a look at the following paper: Bayesian automatic relevance determination algorithms for classifying gene expression data, and try to estimate the probability of common feature occurring together over a given number of iterations (i.e. in order to estimate an optimal $n$, as opposed to picking one arbitrarily).

Ignoring the presented two algorithms in the paper, I focused on the following: $\mathbf{P}(N_c=n_c|n,n_1,n_2) = \frac{\binom{n_a}{n_c}\binom{n - n_a}{n_b - n_c}}{\binom{n}{n_b}}$, where $n_a = \max(n_1, n_2)$, and $n_b = \min(n_1, n_2)$

So basically, my approach is to perform the following over a number of iterations:

  1. Put together the training sets of the two classes (and keep the indices of the examples of each one of them)
  2. Split the resulting training set in half and keep working separately on each half (subset)
  3. Determine the m (which is not necessarily equal to n) most prominent features using the Standard score, and keep track of the frequency of occurrence of each

At the end, I calculate $N_c$ for $c$ from 1 to my feature vector length (i.e. 2500) and thus I get an optimal number for $n$ (the reduced number of features, which if picked arbitrarily could lead to missing out important features, or including not so prominent ones).

So here comes my confusion - is my approach so far correct at all? And if yes - what do I do with the $N_c$ which has the highest probability. Do I put together the $N_c$ most prominent features (from each subset), based on the accumulated values of the frequency vectors, or I should do something entirely different (e.g. like taking the best $N_c$ features from a single scoring - using the Standard score - over the entire training set)?

  • 1
    $\begingroup$ Can you elaborate more on how did u use the z-score to sort the variables? $\endgroup$
    – soufanom
    Commented Dec 8, 2012 at 8:08
  • $\begingroup$ The scoring method is irrelevant to my question above. I gave the Z-score as just an example - it can basically be any other scoring method (that is why I said that I ignored the suggested two algorithms for feature scoring). The important part is calculating the probabilities of a number features occurring together, and then choosing each ones exactly based on the scoring method employed (which once again, is not really important for the discussion here). $\endgroup$
    – User3419
    Commented Dec 8, 2012 at 10:36

1 Answer 1


Basically, two approaches are mentioned in your post:

1- Using single FS metric over the entire training dataset 2- Partitioning data and using the FS metric over every split

In the second case, frequency of the feature among the splits is used to select the final set of features.

Your proposed framework (or the 2nd approach) can be viewed as applying feature selection in a cross-validation setting. I do not think that the question is about the correctness of the proposed idea. Rather, it is more about the proper design and optimized results.

The idea proposed can be used to select a subset of features. However, you may need to split the data into more parts instead of only two halfs which also, can be fine. You may experiment with different number of splits.

Finally, if you prefer or if you are required to use cross-validation, you may go with the second approach. Otherwise, just apply feature selection over the training data set only.

  • $\begingroup$ First, thanks very much for your reply, and for the summarisation of my ideas in a well-structured way (using proper terminology). I should say here that I am not required to use any particular method, rather the question was in regard to a recent suggestion that I received - to try incorporate the approach presented in the paper (after the introduced two algorithms) to my particular problem, explained above. Hope that clarifies things a bit. $\endgroup$
    – User3419
    Commented Dec 8, 2012 at 20:22
  • $\begingroup$ You may go with both approaches and conclude about the best approach based on performance, generalization and efficiency. $\endgroup$
    – soufanom
    Commented Dec 9, 2012 at 8:13
  • $\begingroup$ The problem is that I don't think either of them is correct with respect to the cited paper, since I am getting ridiculously low probabilities (i.e. less than 10^(-30)), whilst with random data the highest probability I get on the average is about 0.20. $\endgroup$
    – User3419
    Commented Dec 9, 2012 at 17:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.