-1
$\begingroup$

I have a small 20k (176 variables) test dataset. I want to create a random bigger dataset for testing, using the rows (data) and columns (names) from the small dataset.

Currently I have:

real_data <- read.csv("somefilehere.csv")

# load one fake row so assignment of column names won't *ahem* complain
test_data <- data.frame(real_dat[1,])
colnames(test_data) <- colnames(real_data)

So, this sets my test_data dataframe to have the same column names as the real data.

How can I run through all the column names and then randomize test_data with existing values from real_data?

The following code works perfectly for randomizing a sample from the existing row:

sample(unique(real_data$some_variable_here),random_count,replace=T)

How do I use iteration of the columns, together with the sample function to populate test_data automatically? Doing manual assignments for every variable would take forever, maybe there is a function in R that I'm missing as a 3 week beginner?

I'm working with R, knitr and LaTeX for processing and would like to use the test_data set for use in my statistical tests and graphs (to see if my document scales well with larger numbers, and my legends still look ok etc).

[edit] An example with randomly picked R dataset.

Dataset: warpbreaks - load with data(warpbreaks)

dataset

Currently it has 3 already defined columns and 54 observations.

How do I create a new dataset called warpbreaks_huge, that has the same column names, but say 25 million observations? Also, from the existing dataset we can see that the tension column has default values L, M and H. The wool column has default values of A and B.

I guess my question is, how do you create a randomized bigger populations programmatically?

Apologies for my naive phrasing. I'm new at this environment.

$\endgroup$
  • 2
    $\begingroup$ This would take a random sample of data.frame rows: test_data <- real_data[sample(seq_len(nrow(real_data)),random_count,replace=TRUE),]. Is that what you need? Or do you really want to sample differently for each column and only unique values? That seems dubious from a statistical pov as rows usually correspond to cases. $\endgroup$ – Roland Sep 23 '13 at 13:17
  • $\begingroup$ No @Roland, I don't want to use a random sample of the existing dataset. The whole point is to generate random data from existing row data in the real data set. The whole intent is to create 'dubious' data here (but from a seed set - so given columns still look the same in my graphs), but the dataset is bigger. With this, I can see if graphs still look the same with random different datasets, because this is a write once report, with a yearly new datadump (why I use LaTeX + knitr). I want the report to be robust when new data comes in, and for that, I need to randomize the dataset to check. $\endgroup$ – StatsBeginner Sep 23 '13 at 13:27
  • $\begingroup$ An example would be - you have a column real_data$countries - so you get country names from this, but randomize to create more rows in dataset for testing. I might have another variable real_data$countries, it has existing values of male, female and other - so I read from this column, and then randomize new data from this. I hope this is clearer? $\endgroup$ – StatsBeginner Sep 23 '13 at 13:29
  • $\begingroup$ Please give an example with toy data. I don't really follow. Possibly you are looking for sapply/lapply. $\endgroup$ – Roland Sep 23 '13 at 13:31
  • 1
    $\begingroup$ Normally you would do resampling using something like the code in my first comment, i.e. bootstrap resampling. I don't understand why this isn't satisfactory for your use case. $\endgroup$ – Roland Sep 23 '13 at 14:19
1
$\begingroup$

To get this answered:

What you want to do is essentially bootstrap resampling: You sample with replacement in order to create a bigger sample (and possibly infer properties of the distribution). From a statistical point of view it's important that rows normally equal cases.

You can do the resampling like this:

test_data <- real_data[sample(seq_len(nrow(real_data)), random_count, replace=TRUE),]

This code creates a vector of row numbers, samples this vector with replacement and subsets the data.frame with the result.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.