0
$\begingroup$

I have data with 5,000,000 records and many categorical variables of it have more than 3 categories. For example(here is the Python Pandas code)

>>> df_train_set.PWD_STAT.unique()
[ 0.  1.  3.  2.]

>>> df_train_set.CUST_CLASS_XL.unique()
[500101 620101 540202   5001 560301 600103 610201 610103 540101 570101
 500102 590208 630112 510201 550104 600104 530101 640101 560102 580106
 590202 630107 530102 640105 530201 580101 570201 540201 610102 550102
 630114 560104 520101 610101 510101 560101 630108 630106 550101 610204
 610202 510105 580103 520103 510103 640103 590201 510104 520201 550301
 600101 520102 560103 630101 590102 630118 570102 590209 590101 600102
 560105 530402 630117 640102 630105 580104 630109 560201     50 600301
 630113 580102 590207 590206 630115 630116 610501 630102 630111 590104
   6301   5401 630103   6201 630110 640104   5801   5402]

In the above output, each value in the list means a category.

Since here the above variables are not binary variables. And I search the internet that maybe I can transform these variables into dummy variables. What is the best way to process these columns of categorical variables?

$\endgroup$
  • $\begingroup$ Are you asking for code to create dummy variables? $\endgroup$ – Arun Jose Dec 6 '17 at 7:36
  • $\begingroup$ @ArunJose, No, I know how to get dummy variables. What is the best way to process these columns of categorical variables? $\endgroup$ – GoingMyWay Dec 6 '17 at 7:46
  • $\begingroup$ Are you attempting some form of classification? $\endgroup$ – Arun Jose Dec 6 '17 at 7:47
  • $\begingroup$ @ArunJose, Yes. Since creating dummy variables will make the dimension booming. And I don't know the best methods to process categorical variables. $\endgroup$ – GoingMyWay Dec 6 '17 at 7:50
0
$\begingroup$

Two ways that are commonly used:

1) Create dummy variables only for categories that occur frequently enough - this way you ignore categories which are rare and contribute very little information.

2) Group together categories which have similar response rates - This again reduces number of dimensions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.