2
$\begingroup$

Is there a method for creating a large data set from a smaller one?

I have a data set of anthropometric variables (e.g. stature, leg length, arm length and so on)

So I have 7 variables and 1774 samples. From this data I would like to create a much larger data set of the same 7 variables but with 100,000 samples.

I know there is an "almost" linear relationship between stature and the other body-measurements but I want random variation in my data.

I am using Python, and I have looked at PCA, multivariate analysis and stuff like that. Here's an illustration of my problem:

enter image description here

Edit: Here is additional information. The data contains 7 variables that describes the dimensions of 1774 people. I want to create dimensions of random 100,000 people who are different from each other but still remains realistic.

I tried this method, but cannot figure out how or whether it is possible to generate realistic data using this.

I would assume that the variables are normal distributed. I know that variable "2-7 correlates" well with "variable 1" which is stature.

|cite|improve this question
$\endgroup$
  • $\begingroup$ What is the purpose of creating the larger dataset? Will you use it for inference or for something else? $\endgroup$ – MånsT Mar 31 '15 at 11:27
  • 1
    $\begingroup$ I would turn this round. If there were a good reasoned way to do this, it would need to be in every introductory course and text, as having a small sample and wanting a bigger one remains one of the most common needs. If and only if you have grounds for thinking that your data are close to a brand-name distribution does it make sense to simulate from that distribution. But then it is not your dataset legitimately made bigger, but just a bigger dataset constructed by analogy. $\endgroup$ – Nick Cox Mar 31 '15 at 11:29
  • $\begingroup$ The data describes the dimensions of 1774 persons. I want to generate dimensions for 100,000 different sized persons. $\endgroup$ – Kasper Rasmussen Mar 31 '15 at 12:52
  • 1
    $\begingroup$ Yes I would like to generate fake data, but it has to include the correlation between body-segment-lengths, to insure realistic people. $\endgroup$ – Kasper Rasmussen Mar 31 '15 at 18:49
  • 1
    $\begingroup$ @KasperRasmussen if so, check my edit - I provided an additional link to thread describing how to do it with examples and code in R. I don't go into further detail so not to duplicate it. $\endgroup$ – Tim♦ Apr 1 '15 at 6:26
3
$\begingroup$

If you oversample your sample you would have the same data. Say your sample is: $1,2,3$ and you sample $2N$ values out of it and get: $1,1,2,2,3,3$ - now if you compute some statistic on it (say a mean), then you'll get the same result for both since they contain the same observations appearing with the same probability. Thing that could change is p-value because of sample size, but this would be p-cheating.

If you want to approximate the unknown distribution of your data, then one thing that could be done is to use bootstrap, i.e. sample with replacement $N$ out of $N$ cases $R$ times and use this data to approximate the distribution of your sample. If there is linear relation and you are interested in learning about the distribution of the "variation" around it, then you could use bootstrap in different fashion: fit the linear model and then sample the residuals, so to approximate the distribution of residuals.

You could also conduct a simulation that generates a sample that resembles your data. For example, to generate correlated variables given some covariance matrix you can sample from Multivariate Normal distribution - for learning more on this check this thread.

If the sample size is a problem, then consider using a statistical method that is robust to small sample size - for example Bayesian simulation-based statistical methods often work quite well with small samples.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ If they sub-perturb, make small perturbations of each point so that it is substantially within its distribution, that can help. They could randomly pick two points and average them, or weighted average them, and use that. There are about 10 unique methods to estimate central tendency and any of them could be used, not just the classic mean. You could randomly sample multiple points and get their geometric mean. One of the risks there is if you sample between two modes, then you put something "out in the boonies" that shouldn't be there. $\endgroup$ – EngrStudent - Reinstate Monica Mar 31 '15 at 11:24
  • $\begingroup$ Hey @Tim. I have tried to create a co-variance-matrix from my data so I could use the Cholesky decomposition. I tried to use numpy.cov(data), but some of the eigenvalues were negative. Is there another method, LU decomposition? $\endgroup$ – Kasper Rasmussen Apr 9 '15 at 20:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.