Predictions with small datasets I started learning Machine learning a while ago and now I face my first real world example. The problem is I have small data set (less then 50 rows) and about 200 features to predict one continuous value (regression problem).
The main idea is to create a model, and then play with the features until I get the the predicted value close to the value that I want. Then I make an experiment and hopefully the value is close to predicted.
I read some articles about small data in Machine learning, so I know to use simple models (linear regression, random forest), use feature selection, use K-Fold for cross-validation, remove outliers, maybe generate synthetic data, use Ensemble learning, Regularization.
I would like to know what are your recommendations, can I solve the problem any other way (maybe not even machine learning?), and how to improve accuracy. If you have any sources on this that would be also great. I only used google till now.
 A: There are some challenges when you have more variables than observations.  Linear models (e.g. Linear regression and logistic regression) won't be able to be fit on the raw data, so you will either need to do variable selection or project the data onto a lower dimensional space a la PCA or similar methods.  Additionally, you could use a penalized method like Ridge or Lasso to fit the model with all variables.  I'd anticipate you'd need a large penalty, which may mean that most variables are 0 if not close to 0.  Maybe that is a good thing for you, maybe not.
The problem is compounded when you take a traditional train/test split with the data.  If you held out even 30% of your data, that is 15 samples of your 50.  That means you have very little data to train on and even less to evaluate the performance out of sample.  In this scenario, I would recommend validation with the bootstrap and estimate optimism of the training error so that you can use all the data for training.
Above all else, my recommendation would be to get more data.  Fifty samples is not a lot, and if your intent is to determine which variables are more important over others, I think you're going to make errors just based on how many variables you have and how few observations you have.
A: To drop a few keywords that are worth looking into: You can do some reading on factor analysis (i.e., reducing the dimension of your feature set) or look into the literature on large P, small N problems that are for instance very common when working with microarray data. Depending on your specific type of problem, Bayesian Model Averaging may also be helpful. Generally, there is a large and active literature on (Bayesian) variable selection that may have some useful hints.
An excellent tool that is always worth a try when it comes to out-of-sample predictions is the BART framework.
