I am trying to make a prediction on a new dataset via LendingClub's rest api. I also have historical data from them which I am using to create the model.

I have split the data into train/test sets and then predicted on the entire historical dataset with no issues.

My issue is regarding imputing the mean, which is done on the historical dataset without any problems because there's so much historical data compared to the api. The api data may have very few loans so sklearn's imputer function will drop the columns with all NaNs in it so my prediction will fail because it expects the same number of columns as my model.

I can only think of two ways to solve this, which is to drop the same api missing columns in historical as well but that might remove useful data. Alternatively, I could try to merge the tiny api data with the huge historical dataset, but that might be very time consuming? Any thoughts?


I'm assuming that you are calling FIT on the training data, serializing the model, and then only calling TRANSFORM on the evaluation/api data (i.e your not trying to FIT on the small/api data which you shouldn't do anyway :). Here's a little test that worked for me.

In [1]: train_df = pd.DataFrame(data) 
    a   b   c
0 foo 1.0 2.0
1 foo 1.2 2.2
2 foo 1.4 2.4
3 foo NaN NaN
4 foo 1.8 2.8

In [2]: imp_mean = SimpleImputer(missing_values=np.nan, strategy='mean') 
In [3]: imp_mean.fit(train_df[['b','c']]) 
In [4]: imp_mean.transform(train_df[['b','c']]) 
  array([[1. , 2. ],
         [1.2 , 2.2 ],
         [1.4 , 2.4 ],
         [1.35, 2.35],
         [1.8 , 2.8 ]])

In [5]: evaluation_df 
    a   b   c
0 foo NaN 2.0
1 foo NaN 2.2
2 foo NaN 2.4
3 foo NaN NaN
4 foo NaN 2.8

In [6]: imp_mean.transform(evaluation_df[['b','c']]) 

array([[1.35, 2. ],
       [1.35, 2.2 ],
       [1.35, 2.4 ],
       [1.35, 2.35],
       [1.35, 2.8 ]])
| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.