2
$\begingroup$

I have training data with 260 observations that have a total of 7 classes. Each observation has 120 features. I applied feature selection based on the Bhattacharyya Algorithm and got the top 40 features for each class.

I have two questions. I have done the feature selection on the whole set of observations and will do the training on some part of the data set (50%) and testing on the left out part of the data set. Is this method okay? Or should feature selection also be performed only on the training data?

Also once I have the top 40 features of each class how do I provide an SVM the selected feature set that has say, features 1,2 and 5 are important for class 1 and feature 1,2 and 6 are important for class 2. I am using matlab as an implementational tool. Thanks in advance.

$\endgroup$
7
  • $\begingroup$ You need to construct a training set with the combined set of features for class 1 and 2 for the training algorithm. In your example this means you need features 1, 2, 5 and 6. In fact, I would initially run an SVM without doing feature selection entirely since SVM is robust against bad features. $\endgroup$ Commented Nov 12, 2013 at 14:17
  • $\begingroup$ I'm not familiar with the Bhattacharyya Algorithm, but if it uses the label information when selecting features (from your discussion it almost certainly does) and you use the entire dataset to do this, your error estimate will be overly optimistic. $\endgroup$
    – alto
    Commented Nov 12, 2013 at 17:39
  • $\begingroup$ @alto What do you mean by label information? Bhattacharyya distance finds the inter class distance while selecting features. Whichever set of features maximizes this distance is considered to be most unrelated. $\endgroup$
    – Sohaib I
    Commented Nov 12, 2013 at 17:43
  • $\begingroup$ @MarcClaesen Thanks. There are around seven classes. If I take the best 40 of each class there is a probability of me selecting all the features. Do you think that is alright? $\endgroup$
    – Sohaib I
    Commented Nov 12, 2013 at 17:48
  • $\begingroup$ My response to your comment turned into an answer. $\endgroup$
    – alto
    Commented Nov 12, 2013 at 18:08

3 Answers 3

2
$\begingroup$

The purpose of splitting your data into a train/test sets is to simulate the real world. You have a bunch of labeled data to train a classifier and want to see how it performs on data that was not used to train it. By using the labels from the test set to perform feature selection you've used information that would not be available, since you're pretending you don't know the labels on the test set, and biased your estimates.

Here's an extreme example. Suppose I use the following feature learning algorithm: map each data point to a binary indicator for the corresponding label. I do this using all the data. Now for any reasonable split of my data, any reasonable classifier is going to be able to figure this relationship out, and get 100% accuracy on the test set. Granted, my algorithm is for feature learning, not feature selection, but I think you'll see the point I'm trying to make.

$\endgroup$
3
  • $\begingroup$ I completely understand your point. Thanks for the answer. Do you then mean that the selection should be done on the training set or that there should be three sets one for training one for testing and a third for selection alone? $\endgroup$
    – Sohaib I
    Commented Nov 12, 2013 at 18:15
  • $\begingroup$ There should be no problem using the training set to do feature selection, besides possibly overfitting, but that is another issue entirely. When designing these types of procedures, the hypothetical forecaster analogy is always something worth keeping in mind. $\endgroup$
    – alto
    Commented Nov 12, 2013 at 18:42
  • $\begingroup$ Thanks! Im a beginner in machine learning mate. Things like hypothetical forecaster analogy go like whooosh! :P Thanks again though for clearing doubts about selection :) $\endgroup$
    – Sohaib I
    Commented Nov 12, 2013 at 18:54
1
$\begingroup$

There are a few issues. Hopefully something will be useful. I should say these are opinions, and can't justify much of this.

(1) Can I use the full dataset for feature selection?: No. This will effect the validity of your test sample estimate. (I think this has been covered on this site before but can't find the thread)

(2) Split sample/Why are people talking about k-fold validation? Your sample is relatively small. Split sample validation is as a result likely to give you an unreliable estimate of your test error. Thus many would recommend estimating test error using a cross validation method such as k-fold (but you could use any of the cross validation methods including bootstrap)

(3) The challenge with cross validation is that - like split sample - feature selection should be done in each cross-validated sample. Some software does this for you, but often is doesn't so this is a challenge with cross validation.

(4) Lots of people do feature selection in the whole sample. I'm not sure how much it really biases your test error. It is really up to you how rigorous you want your endeavor to be.

$\endgroup$
3
  • $\begingroup$ I was thinking about bootstrap and just randomly selecting 50% of my samples as training. Thanks for the reply. If not using k-fold on which set do I perform selection? Also if you read about my question about how to train in the last para of the question I would be much obliged if you could provide me with an answer to that.(Marc replied but I couldnt understand completely) $\endgroup$
    – Sohaib I
    Commented Nov 12, 2013 at 18:12
  • $\begingroup$ A. training and feature selection happen on the same test. Feature selection is conceptually thought of as the first part of model training and not a separate process. B. The benefit of bootstrap (or any cross validation approach) is that you can use the whole dataset for model training and testing. So if you bootstrap you don't need to split your dataset at all. (3) Sorry about the last paragraph - almost never use matlab or SMV - so don't think I can add anything useful. $\endgroup$
    – charles
    Commented Nov 12, 2013 at 18:23
  • $\begingroup$ Thanks. I will try out and report back in a day or two. I have a few doubts about feature reduction techniques like PCA can we move this over to chat if you have some knowledge on that? $\endgroup$
    – Sohaib I
    Commented Nov 12, 2013 at 18:28
0
$\begingroup$

You can use K-fold cross validation. Divide the data-set into k equal sub-sample and use one data-set for testing the model and remaining k-1 sample data sets are used for training. Then cross validation process is then repeated k-times(k-folds).

$\endgroup$
1
  • $\begingroup$ I do know about K-fold cross validation. My question was about feature selection. Is it alright to use the whole data-set for feature selection? $\endgroup$
    – Sohaib I
    Commented Nov 12, 2013 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.