What regularizer to use for small datasets? If I have a sparse dataset with very few points, which regularization scheme should I use?
That is, I have a dataset with only 10 points. Are there regularizers that would help me in this situation?
 A: If you have 10 points, I would say the only regularization "scheme" you'd even want to remotely consider is the fully bayesian one. Look up bayesian regression, bayesian testing, or whatever you're trying to do. There are packages that will help you, but you'll want to be involved in the priors. In many larger data situations, you're not as worried about them (for typical uses of L2 regularization, you've effectively got a prior mean of 0), but with your sample size, they are going to matter. Set them carefully, by thinking about what you know and don't know. You'll get "regularization," but it doesn't have to be the kind that just makes your coefficients smaller. Rather it could "shrink" them closer to a non-zero prior mean value, for instance. 
A: Which regularization scheme to use is not depending on how much data you have. For example, L1, L2 regularization can be used in both big or small data set.
On the other hand, how much you want to regularize is depending on your data size and the complexity of the model. Suppose we use L2 on polynomial fit for $10$ data points. If you want to use $5th$ order model, it is better to set $\lambda$ larger, comparing to you want to fit with $3rd$ order model.
