How to divide feature set for selection and training I have training data with 260 observations that have a total of 7 classes. Each observation has 120 features. I applied feature selection based on the Bhattacharyya Algorithm and got the top 40 features for each class.
I have two questions. I have done the feature selection on the whole set of observations and will do the training on some part of the data set (50%) and testing on the left out part of the data set. Is this method okay? Or should feature selection also be performed only on the training data?
Also once I have the top 40 features of each class how do I provide an SVM the selected feature set that has say, features 1,2 and 5 are important for class 1 and feature 1,2 and 6 are important for class 2. I am using matlab as an implementational tool. Thanks in advance.
 A: The purpose of splitting your data into a train/test sets is to simulate the real world. You have a bunch of labeled data to train a classifier and want to see how it performs on data that was not used to train it. By using the labels from the test set to perform feature selection you've used information that would not be available, since you're pretending you don't know the labels on the test set, and biased your estimates. 
Here's an extreme example. Suppose I use the following feature learning algorithm: map each data point to a binary indicator for the corresponding label. I do this using all the data. Now for any reasonable split of my data, any reasonable classifier is going to be able to figure this relationship out, and get 100% accuracy on the test set. Granted, my algorithm is for feature learning, not feature selection, but I think you'll see the point I'm trying to make.
A: There are a few issues. Hopefully something will be useful. I should say these are opinions, and can't justify much of this.
(1) Can I use the full dataset for feature selection?: No. This will effect the validity of your test sample estimate. (I think this has been covered on this site before but can't find the thread)
(2) Split sample/Why are people talking about k-fold validation? Your sample is relatively small. Split sample validation is as a result likely to give you an unreliable estimate of your test error. Thus many would recommend estimating test error using a cross validation method such as k-fold (but you could use any of the cross validation methods including bootstrap)
(3) The challenge with cross validation is that - like split sample - feature selection should be done in each cross-validated sample. Some software does this for you, but often is doesn't so this is a challenge with cross validation.
(4) Lots of people do feature selection in the whole sample. I'm not sure how much it really biases your test error. It is really up to you how rigorous you want your endeavor to be. 
A: You can use K-fold cross validation. Divide the data-set into k equal sub-sample and use one data-set for testing the model and remaining k-1 sample data sets are used for training. Then cross validation process is then repeated k-times(k-folds).  
