Questions about variable selection for classification, and different classification techniques

Question

I have a question concerning feature selection and classification. I will be working with R. I should start by saying that I am not very familiar with data mining techniques, aside from a brief glimpse provided by an undergraduate course on multivariate analysis, so forgive me if I am lacking in details regarding my question. I will try my best to describe my problem.

First, a little about my project: I am working on an image cytometry project, and the dataset is composed of over 100 quantitative features of histological images of cellular nuclei. All of the variables are continuous variables describing features of the nucleus, such as size, DNA amount, etc. There is currently a manual process and an automatic process for obtaining these cellular images. The manual process is (very) slow, but is done by a technician and yields only images that are usable for further analysis. The automatic process is very fast, but introduces too many unusable images - only about 5% of the images are suitable for further analysis, and there are thousands of nuclear images per sample. As it turns out, cleaning the data obtained from the automatic process is actually more time consuming than the manual process.

My goal is to train a classification method, using R, to discriminate between good objects and bad objects from the data obtained from the automatic process. I have an already classified training set that was obtained from the automatic process. It consists of 150,000 rows, of which ~5% are good objects and ~95% are bad objects.

My first question deals with feature selection. There are over 100 continuous explanatory features, and I would like possibly get rid of noise variables to (hopefully) help with the classification. What methods are there for dimensionality reduction with the goal of improving classification? I understand that the need for variable reduction may vary depending on the classification technique used.

Which leads to my second question. I have been reading about different classification techniques, but I feel that I cannot adequately determine the most suitable method for my problem. My main concerns are having a low misclassification rate of good objects over bad objects, and the fact that the prior probability of the good objects is much lower than the prior probability of the bad objects. Having a bad object classified as good is less of a hassle than recovering a good object from the pool of bad objects, but it would be nice if not too many bad objects were classified as being good.

I have read this post and I am currently considering Random Forests as per chl's answer. I would like to explore other methods as well, and I would like to collect the suggestions of the good people here at CV. I also welcome any readings on the subject of classification that may be helpful, and suggestions for R packages to use.

Please ask for more details if my post is lacking in details.

Answer 1

Feature selection does not necessarily improve the performance of modern classifier systems, and quite frequently makes performance worse. Unless finding out which features are the most important is an objective of the analysis, it is often better not even to try and to use regularisation to avoid over-fitting (select regularisation parameters by e.g. cross-validation).

The reason that feature selection is difficult is that it involves an optimisation problem with many degrees of freedom (essentially one per feature) where the criterion depends on a finite sample of data. This means you can over-fit the feature selection criterion and end up with a set of features that works well for that particular sample of data, but not for any other (i.e. it generalises poorly). Regularisation on the other hand, while also optimising a criterion based on a finite sample of data, involves fewer degrees of freedom (typically one), which means that over-fitting the criterion is more difficult.

It seems to me that the "feature selection gives better performance" idea has rather passed its sell by date. For simple linear unregularised classifiers (e.g. logistic regression), the complexity of the model (VC dimension) grows with the number of features. Once you bring in regularisation, the complexity of the model depends on the value of the regularisation parameter rather than the number of parameters. That means that regularised classifiers are resistant to over-fitting (provided you tune the regularisation parameter properly) even in very high dimensional spaces. In fact that is the basis of why the support vector machine works, use a kernel to tranform the data into a high (possibly infinite) dimensional space, and then use regularisation to control the complexity of the model and hence avoid over-fitting.

Having said which, there are no free lunches; your problem may be one where feature selection works well, and the only way to find out is to try it. However, whatever you do, make sure you use something like nested cross-validation to get an unbiased estimate of performance. The outer cross-validation is used for performance evaluation, but in each fold repeat every step in fitting the model (including feature selection) again independently. A common error is to perform feature selection using all of the data and then cross-validate to estimate performance using the features identified. IT should be obvious why that is not the right thing to do, but many have done it as the correct approach is computationally expensive.

My suggestion is to try SVMs or kernel logistic regression or LS-SVM etc. with various kernels, but no feature selection. If nothing else it will give you a meaningfull baseline.

Answer 2

On dimensionality reduction, a good first choice might be principal components analysis.

Apart from that, i don't have too much to add, except that if you have any interest in data mining, I strongly recommend you read the elements of statistical learning. Its both rigorous and clear, and although I haven't finished it, it would probably give you much insight into the right way to approach your problem. Chapter 4, linear classifiers would almost certainly enough to get you started.