Good classifiers for small training sets What are good classification methods in the case of continuous independent variables (features) and a small training set (particularly where the number of training examples is approximately equal to the number of independent variables)? Here, small means about 50. I am particularly interested in being able to know which variables are “significant”. Ideally, I'm looking for a method for which the training step is computationally efficient; I care much less about the computational cost for the actual classification task.
 A: You want to keep your model as simple as possible so it won't overfit. This usually means making simple assumptions about the distribution the data comes from.
Some possibilities are Naive Bayes, Logistic Regression, some type of decision tree, maybe linear SVM (without playing with the external parameters too much).
Also, you should try to have a very small number of features. You can try various feature selection methods, but if you want to learn about the importance of the original features, try not to distort the feature space (e.g. no PCA).
A: First of all, you may want to have a look at the Elements of Statistical Learning. They discuss variable selection as well as different regularization techniques in chapter 3 (never mind it being about regression).
If you think your variables are basically not correlated, and should go either into the model or not, then you may want to have a look at random forests. They try to cope with the small sample size problem by building a large number of models from slightly varying subsets of the data (subsetting both cases and variates). In addition, they can tell you how many decision trees use which variate, which could help your variable selection.
However, if you think your variates may be correlated, methods like PCA-LDA or PLS-LDA may be more appropriate. If you chain them correctly, you can even derive coefficients that tell you how much of the original variates goes into what LD function. (You can ask me for R code, if that helps). 
I'd go for LDA instead of logistic regression here, as LR tends to need more training cases. 
