Determining conserved features using a Bayesian approach

Question

I would like to perform some sort of binary classification, and my data set consists of 100 examples (for each class), which are vectors with 2500 elements. Ideally, I would like to determine which the most important features are (i.e. best discriminating between those two classes) and reduce the number of features to a more reasonable dimension, e.g. 50.

I decided to start off with a naïve approach, using the Standard score (by only determining the indices of the most prominent features, but without modifying the actual values), which obviously does not take into account the distribution of values per each class. What I basically do is I calculate the score and order in descending order of prominence, and store the indices. Then I just keep the first $n$ (say 50, as suggested above) of them and extract only those features.

Just recently, it was recommended to me to take a look at the following paper: Bayesian automatic relevance determination algorithms for classifying gene expression data, and try to estimate the probability of common feature occurring together over a given number of iterations (i.e. in order to estimate an optimal $n$, as opposed to picking one arbitrarily).

Ignoring the presented two algorithms in the paper, I focused on the following: $\mathbf{P}(N_c=n_c|n,n_1,n_2) = \frac{\binom{n_a}{n_c}\binom{n - n_a}{n_b - n_c}}{\binom{n}{n_b}}$, where $n_a = \max(n_1, n_2)$, and $n_b = \min(n_1, n_2)$

So basically, my approach is to perform the following over a number of iterations:

1. Put together the training sets of the two classes (and keep the indices of the examples of each one of them)
2. Split the resulting training set in half and keep working separately on each half (subset)
3. Determine the m (which is not necessarily equal to n) most prominent features using the Standard score, and keep track of the frequency of occurrence of each

At the end, I calculate $N_c$ for $c$ from 1 to my feature vector length (i.e. 2500) and thus I get an optimal number for $n$ (the reduced number of features, which if picked arbitrarily could lead to missing out important features, or including not so prominent ones).

So here comes my confusion - is my approach so far correct at all? And if yes - what do I do with the $N_c$ which has the highest probability. Do I put together the $N_c$ most prominent features (from each subset), based on the accumulated values of the frequency vectors, or I should do something entirely different (e.g. like taking the best $N_c$ features from a single scoring - using the Standard score - over the entire training set)?

Answer 1

Basically, two approaches are mentioned in your post:

1- Using single FS metric over the entire training dataset 2- Partitioning data and using the FS metric over every split

In the second case, frequency of the feature among the splits is used to select the final set of features.

Your proposed framework (or the 2nd approach) can be viewed as applying feature selection in a cross-validation setting. I do not think that the question is about the correctness of the proposed idea. Rather, it is more about the proper design and optimized results.

The idea proposed can be used to select a subset of features. However, you may need to split the data into more parts instead of only two halfs which also, can be fine. You may experiment with different number of splits.

Finally, if you prefer or if you are required to use cross-validation, you may go with the second approach. Otherwise, just apply feature selection over the training data set only.