Optimal Machine Learning Sample size

Question

I am new in ML area and I want to build a model to score 100,000 people.

I wonder if it is abnormal to build (train/test/validation) the model on a dataset of the same size 100,000? Need to mention that my dataset is highly imbalanced say 95:5 and the aim is to detect the minority class.

In fact, I have observations on 100,000 people with features constructed in a period March to June, and the aim of the model is to predict in the future, say September.

The fact that the model is used to score a population that is quite the same size as the train population makes me feel like the problem is not well-defined. Isn't it? And the fact that the dataset is highly imbalanced and that I would like to learn on minority do increase this feeling.

I heard about optimal sample size in Machine Learning, could someone give me some insights or papers regarding this issue?

Answer 1

We have some different issues.

First one, I don't quite get what do you mean by "imbalanced data set". Also we don't know what do you want to predict. This is important because things change radically. Imagine that I want to predict wether a person is about to find a job or not given some variables (age, education years, marital status and so on). Now, in my 200 size dataset, I only have 10 people employed. This is a problem, as my dataset is very unbalanced, so it can't "learn" how to predict the employment "formula" as I only provided 10 cases out of 200. Sample size is always a nightmare to statisticians, but it depends on what you want to do and how to effectively know how much data you need to do something.

What is always true though (regarding another issue) is the split dataset thing you comment. That is also controversial as I don't think there's a given ratio to split number (I've read papers ranging from 90-10 to 50-50 to even 30-70). Moreover, you should be concerned about the aberration that splitting data implies. If you are "leaving out" observations, you are in fact, losing power. This means that you dataset is now not 200 observations, but let's say (as there's no magic number as we just discussed) 100. Now that's a substantially lower number! And even more if your dataset is "unbalanced", this means that either the test (never validation!) and train sets will have few of that variable, and therefore underfitting/overfittng is now more than plausible. In other words, there are no real reasons to split your data and put you model into a disadvantage, cripping out your dataset only means that you will lose information. It is not worth it to do so just for checking how good (or bad) you did. There are other methods to do this.

My recommendations are:

1. Learn exactly what you want to predict (and why! It might not be the adequate way of doing things.)
2. Learn about the way of doing so and get a basic theoretical foundation (assumptions, ideal scenarios, worst-case uses, red flags, etc) of such model (algorithm for the ML fans)
3. Stay away from data splitting. There are far better (and simpler!) methods (bootstrap, cross-validation, etc.)

Hope this helped

Here, check this resource, as it contain some insight about predictions https://hbiostat.org/rmsc

Good luck

Answer 2

If you have an unbalanced dataset, you should try one of the options to handle like;

• Under-sampling
• Over-sampling
• Cluster the abundant class
• Ensemble different resampled dataset etc.

optimal training/test data split is changing based on how much data you have, for a small dataset %70 - %30 is acceptable but if you have a large dataset %99 - %1 is also well enough. The idea behind that inductive learning algorithm that you use for your task only says the output hypothesis fits your target over the training data. So you keep max the observed data to produce the best-approximated hypothesis to the target function.