What are good classification methods in the case of continuous independent variables (features) and a small training set (particularly where the number of training examples is approximately equal to the number of independent variables)? Here, small means about 50. I am particularly interested in being able to know which variables are “significant”. Ideally, I'm looking for a method for which the training step is computationally efficient; I care much less about the computational cost for the actual classification task.
$\begingroup$
$\endgroup$
3
-
$\begingroup$ Have you tried logistic regression with a LASSO penalty? That's the first thing I'd try. $\endgroup$– Stefan WagerCommented Jul 7, 2013 at 15:30
-
$\begingroup$ I'm just at the stage of gathering ideas, so haven't tried anything, but thanks for the suggestion Stefan. $\endgroup$– ChrisCommented Jul 7, 2013 at 16:21
-
1$\begingroup$ Without using the lasso, and even perhaps using it, the probability of finding the "right" variables is exceedingly low with this sample size. If variables are colinear it's even worse. Make sure that finding features is more important to you than prediction, and avoid classification accuracy as it takes a very large sample size for that improper scoring rule to have sufficient precision. $\endgroup$– Frank HarrellCommented Jul 7, 2013 at 19:46
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
First of all, you may want to have a look at the Elements of Statistical Learning. They discuss variable selection as well as different regularization techniques in chapter 3 (never mind it being about regression).
If you think your variables are basically not correlated, and should go either into the model or not, then you may want to have a look at random forests. They try to cope with the small sample size problem by building a large number of models from slightly varying subsets of the data (subsetting both cases and variates). In addition, they can tell you how many decision trees use which variate, which could help your variable selection.
However, if you think your variates may be correlated, methods like PCA-LDA or PLS-LDA may be more appropriate. If you chain them correctly, you can even derive coefficients that tell you how much of the original variates goes into what LD function. (You can ask me for R code, if that helps). I'd go for LDA instead of logistic regression here, as LR tends to need more training cases.
answered Jul 8, 2013 at 13:06
-
$\begingroup$ Thanks. Some R code demonstrating the chaining you describe would be really useful, if not too much trouble. $\endgroup$– ChrisCommented Jul 8, 2013 at 18:35
-
$\begingroup$ Thanks. I'm not actually using R, and I was simply looking to understand your suggestion better by reading your code, so please don't count on any sensible bug reports :-) $\endgroup$– ChrisCommented Jul 9, 2013 at 9:08
-
$\begingroup$ Oh, OK. Look into pcalda.R or plslda.R. The idea is to train an LDA model in the PCA/PLS scores space. If you make sure that the centering is the same for PCA(PLS) and LDA, then you can calculate "overall" coefficients by matrix-multiplying the PCA rotation matrix (PLS weights) with the LDA coefficients. $\endgroup$ Commented Jul 9, 2013 at 12:38
$\begingroup$
$\endgroup$
2
You want to keep your model as simple as possible so it won't overfit. This usually means making simple assumptions about the distribution the data comes from.
Some possibilities are Naive Bayes, Logistic Regression, some type of decision tree, maybe linear SVM (without playing with the external parameters too much).
Also, you should try to have a very small number of features. You can try various feature selection methods, but if you want to learn about the importance of the original features, try not to distort the feature space (e.g. no PCA).
-
$\begingroup$ Thanks. Naive Bayes was my initial thought, but I don't know about variable selection for that. Any suggestions? $\endgroup$– ChrisCommented Jul 7, 2013 at 19:30
-
$\begingroup$ @Chris one thing you can do for any method, is to test the performance (in cross-validation) for subsets of variables and conclude about the importance of variables from that (for example, leave one variable out on each run. This is not necessarily comprehensive, but it is often useful in practice and is easy to do for small datasets and simple models. $\endgroup$– BitwiseCommented Jul 7, 2013 at 19:44