Can I reproduce the data (for research purposes) if I have a small sample of it? I am involved into a research project and over there I have a small dataset with 35 records of data. As the size of dataset is very small, there is a risk that our machine-learning model become overfit and therefore, I need to have more data.
I am wondering, if I can reproduce these data, given that I guess the distribution of data ?
For example, if I guess that the distribution of the data is normal with mean=5 and std=1.3 then I simply generate 10,000 data and then I will use the reproduced data to train my machine-learning model.
I understand that, as the dataset is small, i might have more than one nominate distribution. For example, the data might be matched with Normal, Beta. Then in this case, how can i know which distribution is the best for me ?
 A: Generally speaking, 35 observations is not enough for ML model. You might consider more simple econometric model, for instance, logistic regression for classification problem.
However, there is one solution. I would suggest bootstrap (sampling from 35 observations with replacement) since bootstrap requires no assumptions about the underlying distribution and reproduces distribution from the small sample in larger one. The problem is that sample with 35 observations only might be not representative for underlying distribution of data, so bootstrapped sample might become useless.
A: Don't do that --- it is very bad
The goal in statistical inference and prediction (including in machine learning problems) is to make a realistic inference that includes recognition of the uncertainty that comes with having a limited amount of data.  If you understate the inherent level of uncertainty in the inference then that is bad.  What you are proposing to do is to generate synthetic values that will overwhelm your actual data and then use this to make your inferences --- if you were to do that then all that will happen is that you will "infer" back to the assumed structure that went in to producing your synthetic data, which will generally be very different to the probabilistic structure of the actual data generating process.$^\dagger$
What you are proposing is a species of logical fallacy.  I have not heard a name for it before, but I call it the "inferential compounding fallacy".  It occures when you take a predicted outcome from a model (or simulated values from an assumed synthetic model) and then feed the outcome back in to your analysis as if it were observed data.  The effect of feeding in unbiased predictions is that you systematically underestimate the uncertainty in your inferences, and the effect of feeding in biased predictions (or synthetic data) is that you systematically bias your inference and underestimate the uncertainty.

$^\dagger$  If we were capable of guessing the probabilistic structure of data generating processes a priori this make the disciplines of statistics and machine learning largely moot.
