7
$\begingroup$

I have a question concerning feature selection and classification. I will be working with R. I should start by saying that I am not very familiar with data mining techniques, aside from a brief glimpse provided by an undergraduate course on multivariate analysis, so forgive me if I am lacking in details regarding my question. I will try my best to describe my problem.

First, a little about my project: I am working on an image cytometry project, and the dataset is composed of over 100 quantitative features of histological images of cellular nuclei. All of the variables are continuous variables describing features of the nucleus, such as size, DNA amount, etc. There is currently a manual process and an automatic process for obtaining these cellular images. The manual process is (very) slow, but is done by a technician and yields only images that are usable for further analysis. The automatic process is very fast, but introduces too many unusable images - only about 5% of the images are suitable for further analysis, and there are thousands of nuclear images per sample. As it turns out, cleaning the data obtained from the automatic process is actually more time consuming than the manual process.

My goal is to train a classification method, using R, to discriminate between good objects and bad objects from the data obtained from the automatic process. I have an already classified training set that was obtained from the automatic process. It consists of 150,000 rows, of which ~5% are good objects and ~95% are bad objects.

My first question deals with feature selection. There are over 100 continuous explanatory features, and I would like possibly get rid of noise variables to (hopefully) help with the classification. What methods are there for dimensionality reduction with the goal of improving classification? I understand that the need for variable reduction may vary depending on the classification technique used.

Which leads to my second question. I have been reading about different classification techniques, but I feel that I cannot adequately determine the most suitable method for my problem. My main concerns are having a low misclassification rate of good objects over bad objects, and the fact that the prior probability of the good objects is much lower than the prior probability of the bad objects. Having a bad object classified as good is less of a hassle than recovering a good object from the pool of bad objects, but it would be nice if not too many bad objects were classified as being good.

I have read this post and I am currently considering Random Forests as per chl's answer. I would like to explore other methods as well, and I would like to collect the suggestions of the good people here at CV. I also welcome any readings on the subject of classification that may be helpful, and suggestions for R packages to use.

Please ask for more details if my post is lacking in details.

$\endgroup$
  • $\begingroup$ I forgot to mention, I am currently exploring the caret package in R. $\endgroup$ – ialm May 10 '11 at 17:44
  • $\begingroup$ @veol caret has a feature selection routine called rfe that you might find useful. You can read more here, but be careful not to use it on algorythms (such as glmnet) that have built-in feature selection. cran.r-project.org/web/packages/caret/vignettes/… $\endgroup$ – Zach May 10 '11 at 20:09
  • $\begingroup$ Do you have much of a budget to explore this problem? Is your dataset one that you can publicly share? $\endgroup$ – Zach May 10 '11 at 20:10
  • $\begingroup$ @Zach Thanks for the link. As for the second comment, I was actually hired to work on this project (I'm a university co-op student), so I have ample time to work on this problem. As for the dataset, I am not allowed to share it, but I may be able to link to some articles that are publicly available concerning the type of analysis we do on the data if you want to read about it. $\endgroup$ – ialm May 10 '11 at 22:09
  • $\begingroup$ @veol Oh, I see. It should be a good learning experience for you then. It sounded like an interesting analysis, I'd love to see one of the articles. $\endgroup$ – Zach May 11 '11 at 15:47
14
$\begingroup$

Feature selection does not necessarily improve the performance of modern classifier systems, and quite frequently makes performance worse. Unless finding out which features are the most important is an objective of the analysis, it is often better not even to try and to use regularisation to avoid over-fitting (select regularisation parameters by e.g. cross-validation).

The reason that feature selection is difficult is that it involves an optimisation problem with many degrees of freedom (essentially one per feature) where the criterion depends on a finite sample of data. This means you can over-fit the feature selection criterion and end up with a set of features that works well for that particular sample of data, but not for any other (i.e. it generalises poorly). Regularisation on the other hand, while also optimising a criterion based on a finite sample of data, involves fewer degrees of freedom (typically one), which means that over-fitting the criterion is more difficult.

It seems to me that the "feature selection gives better performance" idea has rather passed its sell by date. For simple linear unregularised classifiers (e.g. logistic regression), the complexity of the model (VC dimension) grows with the number of features. Once you bring in regularisation, the complexity of the model depends on the value of the regularisation parameter rather than the number of parameters. That means that regularised classifiers are resistant to over-fitting (provided you tune the regularisation parameter properly) even in very high dimensional spaces. In fact that is the basis of why the support vector machine works, use a kernel to tranform the data into a high (possibly infinite) dimensional space, and then use regularisation to control the complexity of the model and hence avoid over-fitting.

Having said which, there are no free lunches; your problem may be one where feature selection works well, and the only way to find out is to try it. However, whatever you do, make sure you use something like nested cross-validation to get an unbiased estimate of performance. The outer cross-validation is used for performance evaluation, but in each fold repeat every step in fitting the model (including feature selection) again independently. A common error is to perform feature selection using all of the data and then cross-validate to estimate performance using the features identified. IT should be obvious why that is not the right thing to do, but many have done it as the correct approach is computationally expensive.

My suggestion is to try SVMs or kernel logistic regression or LS-SVM etc. with various kernels, but no feature selection. If nothing else it will give you a meaningfull baseline.

$\endgroup$
  • $\begingroup$ Thank you for your detailed answer. I am not familiar with support vector machines, but I am eager to try it. After a quick read on wikipedia, I am trying linear SVM with the default settings in R on a random subset of my data (it was taking a while with the whole dataset). Do you recommend any good articles or readings for me to familiarize myself with SVMs? $\endgroup$ – ialm May 10 '11 at 19:20
  • 1
    $\begingroup$ @veol A linear SVM is a good place to start, but don't just use the default settings, the key to getting good performance from an SVM lies in choosing a good kernel and in carefully optimising the hyper-parameters (the regularisation parameter and any kernel parameters). This can be done by minimising the cross-validation error or a bound on generalisation performance (e.g. the "radius-margin" or "span" bounds. I don't use R, but any decent SVM package should automate this procedure via gradient descent or grid search. $\endgroup$ – Dikran Marsupial May 11 '11 at 6:49
  • 1
    $\begingroup$ @veol As to sources of information, as well as the two key papers on SVMs (Boser Guyon and Vapnik and Cortes and Vapnik), there are many good textbooks that cover this sort of thing, such as "learning with kernels" by Scholkopf and Smola, "kernel methods for pattern recognition" by Shawe-Taylor and Christianini or "pattern recognition and machine learning" by Bishop. Best of luck with your project! $\endgroup$ – Dikran Marsupial May 11 '11 at 6:51
  • $\begingroup$ (+1) nice explanation of the basics. Again where a moment where I'd like to add an answer to my favorites (instead of the whole question). $\endgroup$ – steffen May 11 '11 at 7:36
  • $\begingroup$ Thanks for the great answer, I learned a lot. How do I do kernel logistic regression in R? $\endgroup$ – Zach May 14 '11 at 19:48
6
$\begingroup$

On dimensionality reduction, a good first choice might be principal components analysis.

Apart from that, i don't have too much to add, except that if you have any interest in data mining, I strongly recommend you read the elements of statistical learning. Its both rigorous and clear, and although I haven't finished it, it would probably give you much insight into the right way to approach your problem. Chapter 4, linear classifiers would almost certainly enough to get you started.

$\endgroup$
  • $\begingroup$ Alas, PCA was the first thing I attempted to reduce the dimensionality of the data. It did not provide meaningful results, and I was told to be wary of creating transformations that may only be applicable to the training set and not to the future data. Thank you for the book recommendation - though I am familiar with LDA, there are other methods that may prove to be useful. I will see if there is a copy in my building, or at the university library. $\endgroup$ – ialm May 10 '11 at 19:23
  • 1
    $\begingroup$ There's a PDF copy available for download at the link above if that helps. $\endgroup$ – richiemorrisroe May 11 '11 at 8:44
  • $\begingroup$ Ah, hadn't noticed that. I went straight to the table of contents. Reading this feels like a good place to start. $\endgroup$ – ialm May 11 '11 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.