2
$\begingroup$

I have a large data set of about 1.8 million rows with 80 variables. I would like to find a good technique (code or package) in R that can reduce the amount of training data without damaging the representation of the original data too much. I'm going to use this data set for two purposes:

  1. Predicting a binary outcome ( rows that have "true" as a value are only 1.5% of the data).
  2. Predicting a continuous variable.

Any Idea what technique in R can help with this issue?

$\endgroup$
  • $\begingroup$ For the binary outcome variable it would be better to oversample the 1.5%-stratum. That is the idea behind case-control methods: stats.stackexchange.com/questions/132709/… and logistic regression is still valid. When you know the oversamplig ratio you can use it to correct the intercept, all other parameters are correctly estimated. $\endgroup$ – kjetil b halvorsen May 14 '17 at 15:23
2
$\begingroup$

The function createDataPartition of the caret package is designed for random sampling while balancing the distribution of the critical variable.

Here's an example:

set.seed(12)
binary_var <- rbinom(1e6, 1, 0.015)
mean(binary_var)
# [1] 0.015047
numeric_var <- rnorm(1e6)
# [1] -0.001466279

library(caret)
idx_b <- createDataPartition(binary_var, p = 0.5, list = FALSE)
mean(binary_var[idx_b])
# [1] 0.014926
idx_n <- createDataPartition(numeric_var, p = 0.5, list = FALSE)
mean(numeric_var[idx_n])
# [1] -0.001347959

For more details, have a look at the help page of createDataPartition.

$\endgroup$
2
$\begingroup$

Simple random selection of N rows could be done, given x is your data matrix, by

x.subset = x[sample(nrow(x), N), ]

If you consider reducing the number of columns of predictors, you can do principal component analysis or similar methods. However, all depends on the context.

$\endgroup$
  • $\begingroup$ Thanks for your answer @lambruscoAcido, but how many rows will be sufficient ?, I'll edit my question and add the fact that the "true" outcome of the binary variable is only 1.5% . $\endgroup$ – user49422 Sep 29 '14 at 13:05
  • $\begingroup$ @user49422 - what memory/time requirements do you have? This may determine your sample size. $\endgroup$ – Jonathan Lisic Sep 29 '14 at 14:11
  • $\begingroup$ Hello @Jonathan Lisic, I have 32GB RAM on Win7.I ran few models on 50% of tha data and some of them (like random Forest and flexclust) failed because of memory allocation failure. $\endgroup$ – user49422 Sep 29 '14 at 14:35
  • 1
    $\begingroup$ @user49422, what you are going to have to do is to determine what miss-classification rate or variance you are satisfied with. If this is simply as low as possible give your computing resources then you need to figure out what you can do given your computing resources. So try a few increasing sample sizes, plot out the compute time, memory use, and classification rate as a function of your sample size; then extrapolate from this what a reasonable 'max sample size' given your time/compute constraints would be. $\endgroup$ – Jonathan Lisic Sep 29 '14 at 15:27
2
$\begingroup$

This sounds like you want to take a stratified sample of your data so your sample also contains a 1.5% true outcome on your binary variable. There are already a few solutions to this on stackoverflow.

As far as the sufficient number of rows, it is impossible to tell without the level of precision (error) you require. For example, if you want the rate of your binary outcome to be precise within 0.1%, you will need a minimum of 1000 rows before you even attain that level of precision. For 0.001% you will need 10,000, and so on.

For the continuous variable you can take a relatively smaller "sample" sample then use the following equation to determine your approximate required sample size.

Required Sample Size = ("Sample" sample size) * (variance of "sample" sample error) * (Z ^ 2)/(Desired variance of error)

Where Z is the number of standard deviations for your desired confidence level (E.g. Z=1.96 for a 95% interval).

$\endgroup$
2
$\begingroup$

From the constructed models consider the model KNN with k = 1, the tree and the randomForest. Build a model by assembling the previous three

Construct an assembled model, that is, for each record, each model makes a prediction for the category. The assembled model must assign to each record the most repeated category of all three. Calculate the correctness of the resulting model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.