4
$\begingroup$

I have a dataframe with around 37,000 rows and 54 columns. Out of these 54 columns two columns namely 'user_id' and 'mail_id' are provided in avery creepy format as shown below:

user_id                                           mail_id       
AR+tMy3H/E+Re8Id20zUIz+amJkv6KU12o+BrgIDin0=      DQ/4I+GIOz2ZoIiK0Lg0AkwnI35XotghgUK/MYc101I=
1P4AOvdzJzhDSHi7jJ3udWv4ajpKxOn4T/rCLv4PrXU=      BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=
OEfFUcsTAGInCfsHuLZuIgdSNtuNsg8EdfN98VUZVTs=      BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=   
1P4AOvdzJzhDSHi7jJ3udWv4ajpKxOn4T/rCLv4PrXU=      EHNBRbi6i9KO6cMHsuDPFjZVp2cY3RH+BiOKwPwzLQs=
CYRcuV0cR0algMZJ1N6+3uKcqi8iu+6tJNzmBbmgN7o=      K0y/NW59TJkYc5y0HUwDeAXrewYT0JQlkcozz0s2V5Q=

After a detailed analysis of my data I figured out that I cannot drop these two columns from my dataframe as they are too importanct for prediction. I can hash these two features but there is one more interesting thing. There are only 2,000 types of user_ids and mail_ids. So doing one hot encoding can help a lot. My question is that if I convert this into one hot encoding using 'get_dummies' method in pandas with sparse=True, will it be memory efficient or is there any other efficient way to do it?

$\endgroup$
4
$\begingroup$

That "creepy" format is just a form of anonymization - in Python you can use base64 lib to b64encode('hi my name is derek') and get aGkgbXkgbmFtZSBpcyBkZXJlaw== as output. You'll notice the similarity to the above.

When I use hashlib and do base64.b64encode(hashlib.sha1('derek').hexdigest()). I get my name hashed and encoded as b64 - likely what you have above. Might be fun to experiment and see if you can b64.decode(user_name) and get anything useful out of it (unlikely since SHA1 and other popular hashes are one-way).

But anyway, on to your point because that was a tangent:

Yeah, you can hash those together and use pandas.get_dummies if you like. I usually use sklearn for this type of thing, and I like to work within that ecosystem more than with pandas. Either will be equal from a memory standpoint - both implementations use the sparse=True param to indicate that they want to use a numpy sparse matrix instead of a full featured numpy array under the hood.

Sparse matrices are as good as it gets for one-hot encoding problems!

$\endgroup$
2
  • $\begingroup$ +1, nice answer. What does OHE mean? $\endgroup$ – Sycorax Aug 31 '16 at 20:02
  • $\begingroup$ One Hot Encoding - vectorizing categorical data as a sequence of 0's and 1's $\endgroup$ – Derek Janni Aug 31 '16 at 21:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.