Are there any papers/books/ideas about the relationship between the number of features and the number of observations one needs to have to train a "robust" classifier?
For example, assume I have 1000 features and 10 observations from two classes as a training set, and 10 other observations as a testing set. I train some classifier X and it gives me 90% sensitivity and 90% specificity on the testing set. Let's say I am happy with this accuracy and based on that I can say it is a good classifier. On the other hand, I've approximated a function of 1000 variables using 10 points only, which may seem to be not very... robust?