2
$\begingroup$

I have a dataset of 20 million rows. it is organized into strata (groups), and I need to sample from them. I need to create a smaller sampled dataset on which I bulid a regression model.

I need to first determine total sample size and then the sample size for each stratum, and then choose the training and test indexes.

How does one implement all of this in R? can anyone provide some links to this?

$\endgroup$

closed as off-topic by whuber Feb 4 '16 at 14:45

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – whuber
If this question can be reworded to fit the rules in the help center, please edit the question.

5
$\begingroup$

For a stratified sample you can use caTools library. If your factor variable is strata and you want 70% of the data as train, the code is

library(caTools)
train_rows = sample.split(data$strata, SplitRatio=0.7)
train = data[ train_rows,]
test  = data[-train_rows,]

Ps: it should be:

library(caTools)
train_rows = sample.split(data$strata, SplitRatio=0.7)
train = data[ train_rows,]
test  = data[!train_rows,]

Now, it should be working

$\endgroup$
  • $\begingroup$ I tried using sample.split but the code gets stuck. It does not run at all. $\endgroup$ – user76170 May 11 '15 at 10:07
  • 1
    $\begingroup$ It is probably very slow with 20 million rows. If you want to check if the code runs at all, try it on a small subsample. $\endgroup$ – lanenok May 11 '15 at 10:26
1
$\begingroup$

In addition to @lanenok's answer, you seem to need to determine the sample size per stratum. One way to do so is to use Neyman's allocation, which is implemented in the stratification package :

library(stratification)

## Example dataset
N <- 1000
id <- c(1:N)
y <- 10*runif(N) # Quantitative variable from which strata are constructed
df <- data.frame(cbind(id,y))

n <- 50 ## Example of sample size

testStrata <- strata.bh(y, c(2,4,7), n, Ls=4)

allocation <- testStrata$nh

In my example, 4 strata are constructed using the quantitative variable y, which range is (0,11) :

  • Stratum 1 : $y \in [0,2]$
  • Stratum 2 : $y \in [2,4]$
  • Stratum 3 : $y \in [4,7]$
  • Stratum 4 : $y \in [7,11]$

In my example, this gives you an allocation computed with Neyman's formula :

allocation = c(9,7,16,18)

Then, you can draw your sample using the package sampling :

library(sampling)

## These lines construct the strata variable for our example
df$strataId <- sapply(df$y, function(x) { 
  if (x<=2) {return(1)} 
  if(x>=2 && x<=4) {return(2)}
  if (x>=4 && x<=7) {return(3)}
   if(x>=7) {return(4)} 
  return(4)  
                    })

sample <- strata(df, c("strataId"), size=allocation)

which finally gives you 50 units, drawn using stratified sampling and the Neyman allocation.

$\endgroup$
  • $\begingroup$ can you pl explain what this line of code does : testStrata <- strata.bh(y, c(2,4,7), n, Ls=4) ? and why do we provide c(2,4,7) as an argument? how do you get that? $\endgroup$ – user76170 May 11 '15 at 15:21
  • $\begingroup$ It computes the Neyman Allocation, assuming your strata were created regarding the values of a variable $y$. In this example, $y$ varies from $0$ to $11$, and 4 strata are defined, using 4 different value intervals of $y$. $\endgroup$ – Antoine R May 11 '15 at 15:58
1
$\begingroup$

I assume all the data which you have is in x

library(caret)
# considering response variable as strata
data_part <- createDataPartition(y = x$Response, 
                                 p = 0.7, list = F)
test <- x[-data_part,] # 30% data goes here
train <- x[data_part,] # 70% here

Hope it helps.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.