0
$\begingroup$

My goal is to encode a 'Education_Level' which have values 'Uneducated', 'High School', 'College', 'Graduate', 'Post-Graduate', 'Doctorate'.

The problem is that if I drop the first column (in this case 'Uneducated'), then I can't differentiate between 'Uneducated' and unknown category.

i.e. 'Uneducated' and unknown category will have equal row '[0. 0. 0. 0. 0.]'.

I was thinking to NOT drop the first column. However, I am worried because of the effect (never had the chance to test it with a dataset from Kaggle)

What I've tried:

  1. Tried to prove that the encoder can't differentiate between 'Uneducated' and unknown category.
education = ['Graduate','High School','Unknown','Uneducated','College','Post-Graduate',
             'Doctorate']
X = pd.DataFrame(data={
    'Education_Level': education + education + education,
    'Age': [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]
})

X_nom = X['Education_Level'].values.reshape(-1,1)

ohc_enc = OneHotEncoder(categories=[
    ['Uneducated', 'High School', 'College', 'Graduate', 'Post-Graduate', 'Doctorate']
], handle_unknown='infrequent_if_exist', drop='first').fit(X=X_nom)

X_encoded_2 = ohc_enc.transform(X=X_nom).toarray()

# print(X_encoded_2)

print(ohc_enc.inverse_transform(X_encoded_2))

Output

[['Graduate']
 ['High School']
 ['Uneducated'] -> This should be [None], instead if become ['Uneducated']
 ['Uneducated']
 ['College']
 ['Post-Graduate']
 ['Doctorate']
 ['Graduate']
 ['High School']
 ['Uneducated']
 ['Uneducated']
 ['College']
 ['Post-Graduate']
 ['Doctorate']
 ['Graduate']
 ['High School']
 ['Uneducated']
 ['Uneducated']
 ['College']
 ['Post-Graduate']
 ['Doctorate']]
Improve this question
$\endgroup$
2
  • $\begingroup$ Why are you not putting the unknown category in a new category? Then you will be able to differentiate it. $\endgroup$
    – user2974951
    Nov 23, 2022 at 9:42
  • $\begingroup$ @user2974951 Ah, sorry. I was doing GridSearchCV yesterday and the encoder throws warning of unknown category, I though this will be a problem if the train set do not have at least 1 row with value 'Unknown'. I was about to give you the problem, it turns out setting the OneHotEncoder(categories=[['Unknown', etc]]) solves the problem. $\endgroup$
    – Jason Rich Darmawan
    Nov 23, 2022 at 9:50

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.