I am an MBA Student taking courses in Statistics and Data Science.

In one of my Data Science courses, we are learning about different types of predictive models that can be used for Classification Tasks - for example, we can use models such as Decision Trees and Random Forests to predict if a specific customer is likely to file a complaint.

Our professor threw us a "curveball" the other day - he told us that when the size of the data becomes too big, the computer will not be able to fully load the data into memory and you will not be able to directly fit these predictive models on your data the same way you do this for smaller datasets such as the Iris dataset. He asked us to brainstorm different ways we might be able to come across these limitations.

We have the following type of problem - we have a large dataset (e.g. 200 million rows - this dataset is stored on an Oracle SQL Server on which only SQL style operations can be performed) which contains historical information about the customer and the purchase made by the customer (e.g. gender of customer, age of customer, the amount spent on the purchase, new customer or existing customer, etc.), and we have also have information on whether the customer made a complaint or not. Since most customers do not make complaints, the ratio of Complaints to Non-Complaints is very low (e.g. 5:95), thus resulting in something called "imbalanced data". For a new purchase made by a customer, our task is to build a model that will predict if this customer will file a complaint.

After spending some time reading on the internet, I came up with the following strategy for this problem:

  • Step 1: Randomly split the dataset into a 70% training set, 20% test set and 10% validation set (I figured out how to take random subsets using SQL commands)

  • Step 2: In the training test, take a Stratified Random Sample (With Replacement) of 1000 rows such that 40% of these rows contain complaints and 60% of the rows do not contain complaints (I figured out how to perform Stratified Random Sampling with SQL commands) - these are just some arbitrary ratios I came up with, they can be adjusted. This dataset of 1000 rows is now small enough that it can be loaded on to the local computer.

  • Step 3: Using the dataset from Step 2), fit a supervised classification model (e.g. with R/python) to this data such as Random Forest (taking into consideration hyperparameter tuning) on the local computer. Save the model and the results of this model.

  • Step 4: Repeat Step 2 and Step 3 many times (e.g. 1000 times).

  • Step 5: To evaluate the performance of this approach, take the first observation from the test set (the test set is "imbalanced"). See what each of of the 1000 models predicts about this observation - the final prediction for this first observation is the majority vote of all 1000 models. Repeat this voting process for all observations within the test set and record the performance.

  • Step 6: Repeat Step 5) with the validation set.

  • Step 7 (Optional): fine tune steps 1-6 as needed.

It seems to me that although such an approach might be lengthy and time consuming - it still in theory might be able to handle the computer limitations and be a better alternative than trying to load the entire 50 million rows in vain. On another note, this seems like a similar approach to Ensemble/Stacking/Bagging Models, but I do not know a lot about this. My general thought is that the use of Stratified Random Sampling both serves as a way to counter the "Class Imbalance Problem" - as well as explores enough random variation in the data such that the resulting model has a chance of successfully recognizing similar patterns in new data. Furthermore, averaging many models together (in a naïve way) might be able to "smoothen out" inconsistencies within the data, but I am not sure about this.

Can someone please tell me if my idea to the prof's question sounds somewhat reasonable - or am I going to look like Homer Simpson in my lecture class after Thanksgiving?

Clarification: We are not allowed to give answers like "get more money and buy more powerful computers" - I think we are supposed to look at how the field of statistics can help us solve this problem (e.g. sampling)

  • 1
    $\begingroup$ I wonder if this is the kind of question that justifies having a Data Science Stack in addition to this one. $\endgroup$
    – Dave
    Oct 9, 2022 at 7:05
  • 1
    $\begingroup$ Have you asked your classmates? from your description it sounds like the teacher was asking about general strategies eg ( running on larger machine or on multiple machines, not loading up all data in memory, running statistics on the database directly) $\endgroup$
    – seanv507
    Oct 9, 2022 at 9:37
  • $\begingroup$ Dave: Why do you think we need another Data Science Stack? $\endgroup$
    – stats_noob
    Oct 9, 2022 at 16:51
  • $\begingroup$ Seanv507: We are learning about situations in which we do not have a budget to buy larger machines/servers. Based on the way the lectures are formatted, I think the prof wants us to think about using "statistical techniques" to fit these models ... for example: random sampling $\endgroup$
    – stats_noob
    Oct 9, 2022 at 16:54
  • 1
    $\begingroup$ I have mixed feelings about the Data Science Stack (which has existed for years). However, data science is not the same as statistics, and the practical considerations about managing data and implementing statistical ideas in constrained computations settings really are distinct from statistics, justifying having such a community. $\endgroup$
    – Dave
    Oct 9, 2022 at 16:56

1 Answer 1


Let's think in terms of bias and variance for a second. Let's assume there is no confounding so that bias and variance refer only to the properties of the model.

By using a sufficiently flexible model like a Random Forest, we reduce our estimate's bias at the cost of variance. But variance can be combatted by using more data, and there will come a point where using more data does not appreciably change the model estimates. Its hard to pinpoint when that happens, but we know that variance should (in principle) decrease like $1/n$ ($1/\sqrt{n}$ if we're talking standard error).

All this to say, you don't need all the data, you just need enough. How do you define enough? Hard to say. One way you could do this is to subsample as you've mentioned and just see how the predicted risks vary sample to sample. This is a direct estimate of the variability in the predicted risk due to sampling. If after dozen samples or so you see the variation in risk is minimal (e.g. is there a qualitative difference between predicting p = 0.49 and p=0.5? It depends) then you can be confident that the size of the sample results in relatively stable predictions. Then, take a simple random sample from your data, fit your model, and evaluate it using whichever method you choose.

I've purposefully left out comments on fixing the relative frequency of cases to non cases to be something other than what is observed naturally. That question has been addressed before, and I don't feel like rehashing it. You can simply search for "imbalanced data" here and a bunch of answers will come up.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.