4
$\begingroup$

I am trying to run a nonparametric regression on a data set that has 1,000,000 observations and 8 covariates. It is clear I do not have the computing power to run this. I wanted to create a representative subsample of the panel and run the regression with less data. The data has over 13,000 individuals for 72 periods. How could I get a representative subsample? How can I pick which individuals to drop? Or periods?

$\endgroup$
6
  • $\begingroup$ What to you plan to do with the subsample? What kind of inference do you want to make from this subsample? How you go about choosing a subsample is driven by the statistical properties you want it to have. This question isn't really about programming so it seems off-topic for Stack Overflow. If you need help choosing appropriate statistical methods to analyze your data, you should try Cross Validated instead. $\endgroup$
    – MrFlick
    Commented Jan 5, 2015 at 23:21
  • $\begingroup$ I am trying to run a nonparametric regression. Since I have too many observations, I want to reduce the data set, take a subsample. With that subsample I hope to get coefficient estimates similar to those of what I would get from the whole data set. Hence, I wanted to know if R had a function, or how could I use R to pick say a sample of 1000 individuals (instead of 13,000), in a way that does not bias the results. $\endgroup$
    – Goose
    Commented Jan 5, 2015 at 23:29
  • 3
    $\begingroup$ If you didn't have R, how would you do this? What's the method for selection you want to use? R doesn't magically analyze data; it implements many well defined algorithms and methods developed by statisticians over the years. If you tell us what method you are trying to perform, then we can point you to an R function that implements it. There's no magic one-size-fits-all unbiased sampling function for all data in any language. $\endgroup$
    – MrFlick
    Commented Jan 5, 2015 at 23:34
  • $\begingroup$ The OP didn't mention anything peculiar about the data set, like a clustered or matched study design. If that's the case, a simple random sample will suffice as in my answer below. $\endgroup$
    – rsoren
    Commented Jan 5, 2015 at 23:55
  • $\begingroup$ @Goose How is your data structured, as in how many rows and columns do you currently have? $\endgroup$
    – rsoren
    Commented Jan 6, 2015 at 0:10

3 Answers 3

2
$\begingroup$

It seems like you do not have a single row for each subject. Let us assume that your dataframe (assuming it is called df) contains a column called ID which is the unique identifier for each subject.

# Fraction of the subjects to sample
sampling_pct = 0.7
# Obtain an array with unique subject IDs
subject_ids = unique(df$ID)
# Sample from the subject ids
sample_subject_ids = sample(subject_ids, round(sampling_pct * length(subject_ids)))
# Get the rows for the sampled subjects
sample_df = subset(df, ID %in% sample_subject_ids)
$\endgroup$
1
  • $\begingroup$ Thanks. I will try this out with a smaller data set and then move to the bigger one once I fully understand it. $\endgroup$
    – Goose
    Commented Jan 6, 2015 at 6:44
1
$\begingroup$

You can randomly sample rows this way:

df[sample(nrow(df), size = 1000, replace = FALSE),].

The sample size of 1000 is arbitrary in my example. You'll want to choose a sample size based on your memory/computation constraints and the statistical power you're willing to lose.

EDIT: This answer only works if your data frame is structured such that each row is one subject.

$\endgroup$
2
  • $\begingroup$ Are you sure you wouldn't rather randomly select subjects among the panel data and their entire duration of follow-up? $\endgroup$
    – AdamO
    Commented Jan 5, 2015 at 23:54
  • $\begingroup$ As mentioned before, my data set is structured as follows: I have 13600 individuals over 72 periods. The data set is a matrix as follows: Y X1 X2 X3 X4... X8, with Y being the dependent variable. The vector Y has the first 72 observations for individual 1, then followed by 72 observations of individual 2 and so on, 13600 times. Same for the covariates X1, X2, etc. So that is 9 columns and 13600*72 rows. I was wondering if there is a direct way I could use to sample by keeping the data with this structure. $\endgroup$
    – Goose
    Commented Jan 6, 2015 at 6:42
1
$\begingroup$

Not a direct answer, but a more general (and, hopefully, still useful) advice. I would recommend you to use specialized packages for panel data analysis in R, such as plm (http://cran.r-project.org/web/packages/plm). A detailed vignette describes various convenient features of the package. Another R package phtt is also interesting and related, but might not be relevant to your situation.

P.S. Recently I have run across the book "Applied Panel Data Analysis for Economic and Social Surveys" (Springer), which you might also find relevant and helpful. Check this discussion as well.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.