I am trying to make a prediction on a new dataset via LendingClub's rest api. I also have historical data from them which I am using to create the model.
I have split the data into train/test sets and then predicted on the entire historical dataset with no issues.
My issue is regarding imputing the mean, which is done on the historical dataset without any problems because there's so much historical data compared to the api. The api data may have very few loans so sklearn's imputer function will drop the columns with all NaNs in it so my prediction will fail because it expects the same number of columns as my model.
I can only think of two ways to solve this, which is to drop the same api missing columns in historical as well but that might remove useful data. Alternatively, I could try to merge the tiny api data with the huge historical dataset, but that might be very time consuming? Any thoughts?