Have been working on a dataset called Dorothea hosted at UCI. It contains 100,000 features. If I use PCA the top 10 components explain only 4.10% of the variance. Having concluded that PCA will not reduce dimensionality in any significant way, I looked at the dataset in a number of different ways. I found that close to 17000 features are duplicates of one other feature (I took into account the entire data ...i.e. training+validation+test). Is it safe to delete these features?

Has anyone else worked on this dataset. I would like to know what approach did they take.

  • $\begingroup$ I'm not familiar with the dataset, but if they are duplicates, I don't see why you should not delete them. $\endgroup$ – mkt - Reinstate Monica Aug 2 '17 at 19:14
  • $\begingroup$ You should never ever ever ever mix up training set, validation set, and testing set! If there are duplicates in your training data, then we need to think: are they here because some situation naturally repeats itself over time? Several identical sequences could be genuine too. Do you have access to the prior art on this data set where people explain how they pre-processed it? $\endgroup$ – Eskapp Aug 3 '17 at 13:32

If you read the UCI data set page well, you will see that:

"DOROTHEA is a drug discovery dataset. Chemical compounds represented by structural molecular features must be classified as active (binding to thrombin) or inactive. This is one of 5 datasets of the NIPS 2003 feature selection challenge."

and a bit after:

"The dataset with which DOROTHEA was created is one of the KDD (Knowledge Discovery in Data Mining) Cup 2001. The original dataset and papers of the winners of the competition are available at: http://www.cs.wisc.edu/~dpage/kddcup2001/.

and after again:

"This version of the database was prepared for the NIPS 2003 variable and feature selection benchmark The original data were modified for the purpose of the feature selection challenge. In particular, we added a number of distractor feature called 'probes' having no predictive power.

Therefore, if you goal is to do classification and not feature selection, download the original data set and not the one that has been voluntarily spoiled with some distractor features for a challenge. If on the other hand you aim at performing feature selection, you should keep the data set as is, as it has been designed this way to this purpose and that research teams have been using it as is.

  • $\begingroup$ I am interested in the feature selection. $\endgroup$ – Shirish Ranade Aug 2 '17 at 21:47
  • 1
    $\begingroup$ Thank you for the effort you have taken. I am interested in the feature selection. That is how I did the PCA first. It did not seem to help much since the top 10 components only explained 4.2% of the variance. I felt a different approach was required. I used VarianceThreshold and found that some of the features which I found were duplicates were included by the VT transform. I am checking again $\endgroup$ – Shirish Ranade Aug 2 '17 at 22:02
  • $\begingroup$ After setting VT(threshold=(.999 * (1 - .999))), I get a transformed set wherein I find that 10896 features are discardable (all values are zero). However it still has a few (apprx 3,000 features that are dupes). Now I wonder why VT does not eliminate them. $\endgroup$ – Shirish Ranade Aug 3 '17 at 2:27
  • $\begingroup$ Would be interesting seing the code you are using. Would be nice to edit your question to add it :) $\endgroup$ – Eskapp Aug 3 '17 at 13:33
  • $\begingroup$ The dataset hosted at UCI is the dataset used for NIPS-2003.Looks like you have good deal of experience in this field. Can you help me understand what the winners of NIPS-2003 (Feature Selection) have said here ? clopinet.com/isabelle/Projects/NIPS2003/Slides/Neal-Zhang.pdf and translate that in terms of sk-learn tools? Thanks – On slide 7 they say "PCA was done using all the data (training, validation, and test)" $\endgroup$ – Shirish Ranade Aug 6 '17 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.