Working with few data examples I have been asked often in some interview, that how we should proceed when we have less data examples(say 50 or 100). What considerations needs to be made while choosing any algorithm.
few points that came into my mind
1.check if the algorithm is flexible(SVM, kNN) and so avoid them as they would over fit the data easily. naïve bayes seems to be a nice choice as it would be more robust even when the data has some noise.
2.Use of regularization
3.Use of PCA or SVD perhaps to reduce the multicollinearity.  
any other things to be considered?
 A: I'd add


*

*look for a model that fits as closely as possible to the application and data generation process 
I work with data from physico-chemical measurements, and there we have certain ideas how the data should behave (for physical and chemical reasons).  


*

*E.g. for my measurements/applications it often makes sense that the data can approximately be described by linear changes in not-too-large numbers of chemical species. This would fit well with a PLS or PCA regularization.  

*Also, the physical nature of the spectra together with the settings of the measurements typically mean that in the end, coefficients for neighbour variates should be correlated (smooth). This agrees better with, say a ridge (or PCA or PLS) regularization than with LASSO. 

*LASSO would agree better with, say, the situation in gene microarray studies where you expect few informative variates among lots of noise-only channels, and where no relation between neighbour tests on the microarray is expected. 


*If the number of cases is small, testing/validation/proving the performance of the obtained model can become the more difficult part compared to training: For training you can restrict the model complexity/regularize according to what you can afford with the given number of cases (and you can be lucky and face an easy problem), but testing depends on the absolute number of test cases (as the model is treated like a black box). 


*

*One conclusion from that is that with too few cases, data-driven optimization (if you cannot measure the performance well, model comparison may become meaningless) may not work at all.  
To cope with that, you could choose a model where hyperparameters can be fixed by external knowledge
E.g. the number of PLS or PCA components may be guesstimated from expectations about the number of chemical species that I expect to play a role.


*The typical problem with too complex models wrt. the available training sample size is instability. 


*

*So measure the (in)stability of the model and/or its predictions

*And consider model aggregation (e.g. bagging) if stability turns out to be a problem.  


