Here is a statistics question which I have been thinking about while working with some of my data. I have a large dataset named "bigbird" (say about a billion rows) and I want to randomly sample a smaller dataset named "smallbird" from it using R. Now, I can easily do this with the following code:
This is great and my theoretical model works to a certain degree. However, I am trying to finetune my model and I realize that I may have hit a theoretical concern as far as the random sampling is concerned. First, imagine that some of the variables inside bigbird are of similar form as follows:
user,observation_no,year A, 5,1998 B,7,2003 A,6,1998 D,1,2010
Essentially, I have users, a observation number (which references a whole different set of variables) and the year in which they made a certain observation. I have 2 issues that need clarifying as follows:
From looking at the overall dataset, it is evident that as my time period progresses (1998-2012), I have more observations and more new users in every year in an exponential fashion. That is, in 2012, there are many, many more new users in the dataset than in year 1998.
Similarly, as my time period progresses, it appears that many, many more observations are being made by users in a given year than in previous years. That is, in 2012, user A may have made 50 different observations as opposed to only 1 observation in year 1998.
I was looking for opinions, discussions and solutions to these 2 issues because I think (and please correct me if I am wrong), that simple random sampling will not take care of these 2 issues. Thanks !