1
$\begingroup$

I am working on a data set with majorly of categorical columns like Customer Name, Part_Name, Shop_ID etc. I tried encoding these columns and converting them into numerical columns using pd.get_dummies(). However, the categories in the data set are exhaustive. There is always a possibility of getting new categories in the test-data or real-time predictions. How to deal with this problem.

For example: My data set contains column name as Customer_Name. Currently Customer_Name contains Pepsi, ITC, Weatherford, Philips.

However, when my model is deployed in real-time, if the new query contains Customer Name as "Samsung".

There will be an error because I am using pd.get_dummies and the feature size is different.

I am planning to deploy this model as a cloud based API. Are there any solutions available?

$\endgroup$
2
  • $\begingroup$ Why are you using 'Name' in a model ? can you elaborate a little on what model is it and for what problem definition ? If its general predictive model - variables like ID , name won't be used in the main train data $\endgroup$
    – Pb89
    Commented Aug 11, 2017 at 5:52
  • $\begingroup$ You need to explain what you are doing with the data. You say "there will be an error". What is the error? What is the code that generates that error? $\endgroup$
    – BrenBarn
    Commented Aug 11, 2017 at 5:52

1 Answer 1

1
$\begingroup$

If there is a possibility to get new unique values in categorical columns, there are several solution:

  • You can create category 'others' and make new values go there. 'Samsung' and other new companies will become 'other';
  • Retrain model so that it can work with new variables;
  • Encode values in a column into numerical following some rules;
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.