# How to deal with missing values in a 300 GB file (more than 1.5 billion data points)? Statistical Insight required [closed]

## Scenario

I'm working on a binary classification problem involving a 300GB dataset. The dataset's interpretability is low due to privacy concerns. All I have is 6 independent columns (features) and a binary dependent column (target variable).

| col_1 | col_2 |...| col_6 |Target|
|-------|-------|---|-------|------|
| 6.64 | 343 |...| 27.43 | 0 |
| 8.63 | 293 |...| 31.92 | 1 |

All independent columns are numeric. Two independent columns contain missing values as follows:
|Column|Missing value proportion|
|-----------|---------|
|col_1 | 0.20 |
|col_4 | 0.15 |
I don't simply want to impute with a mean/median because I feel impute values will be highly under or over-estimated.

## My solution

I came up with the following approach:

1. Perform K-means on the dataset. Turns out optimal k = 3.
2. Divide the dataset into 3 groups based on k-means.
3. Compute mean for col 1 & 4 (i) for each group(j). x_{i,j} where i =1,4 & j = 1,2,3
4. Compute an interval using I_{i,j} = [x_{i,j} - v_{i,j} , x_{i,j} + v_{i,j} where v is standard error, i=1,4 & j=1,2,3
5. Pick a random number from interval I_{i,j} and use it to impute data point belonging to ith column and jth group.

My reasoning says data points within a cluster resemble more with each other compared to data points from another cluster. Therefore, the mean of, let's say, col_1 and group 1 would be a better candidate for imputation within group 1 compared to traditional mean imputation.

## I need two things:

1. What are the potential problems in my solution from a statistical POV?
2. How would you approach it? Considering standard mean imputation is not an option and if you want to go with an ML model ( SVM etc.) for imputation consider the dataset size (increased training time).
• Welcome to Cross Validated! That would you do if your data set had a size more like $3$ GB or $3$ KB? In other words, in what way does the decently large data size affect the problem?
– Dave
Jan 10 at 20:22