Skip to main content
added 683 characters in body
Source Link
ryan132442
  • 381
  • 2
  • 4

I have a dataset where the task is to classify whether someone looking for a new job will leave their current job, based on a number of factors. (dataset: https://www.kaggle.com/arashnic/hr-analytics-job-change-of-data-scientists)

I'm trying to encode the data and one of the things I struggle with is how to decide whether data is categorical, ordinal.

For example, one of the columns is called 'education_level', and has 5 levels, 'phd', 'masters', 'graduate', 'high school', 'primary school'.

Now if I were to encode this naively, then I'd say it was an ordinal variable because a phd is usually better than a masters, etc. However, I'm struggling to decide whether this is in fact the best way, or, is one-hot encoding more suitable.

The reason is because I don't think I'm trying to rank the candidates by suitability, I'm merely trying to predict whether they will leave their current job, so it doesn't really make sense to me to say that a phd is better than any other level of education. It seems more like an 'arbitrary' partition in this context.

Am I right in thinking that one-hot encoding is better in a situation like this? Or am I simply overthinking and should just go with the naive definition?

EDIT: I read the answer to this question: What is an example of how ordinal data can yield bad classification results in contrast with one-hot encoding?

and have tried to apply it to my problem.

So by using ordinal encoding, I would be enforcing the relationship

$$\hat{y}_{phd} > \hat{y}_{mas} > \hat{y}_{grad} > \hat{y}_{high} > \hat{y}_{prim}$$.

Since I have a classification problem, I'm modelling probabilities, so the implication is that I'm enforcing the constraint that phd's have a higher probability of leaving their job without knowing whether that is true or not. Therefore, I need to use one-hot encoding.

Is this logic sound?

I have a dataset where the task is to classify whether someone looking for a new job will leave their current job, based on a number of factors. (dataset: https://www.kaggle.com/arashnic/hr-analytics-job-change-of-data-scientists)

I'm trying to encode the data and one of the things I struggle with is how to decide whether data is categorical, ordinal.

For example, one of the columns is called 'education_level', and has 5 levels, 'phd', 'masters', 'graduate', 'high school', 'primary school'.

Now if I were to encode this naively, then I'd say it was an ordinal variable because a phd is usually better than a masters, etc. However, I'm struggling to decide whether this is in fact the best way, or, is one-hot encoding more suitable.

The reason is because I don't think I'm trying to rank the candidates by suitability, I'm merely trying to predict whether they will leave their current job, so it doesn't really make sense to me to say that a phd is better than any other level of education. It seems more like an 'arbitrary' partition in this context.

Am I right in thinking that one-hot encoding is better in a situation like this? Or am I simply overthinking and should just go with the naive definition?

I have a dataset where the task is to classify whether someone looking for a new job will leave their current job, based on a number of factors. (dataset: https://www.kaggle.com/arashnic/hr-analytics-job-change-of-data-scientists)

I'm trying to encode the data and one of the things I struggle with is how to decide whether data is categorical, ordinal.

For example, one of the columns is called 'education_level', and has 5 levels, 'phd', 'masters', 'graduate', 'high school', 'primary school'.

Now if I were to encode this naively, then I'd say it was an ordinal variable because a phd is usually better than a masters, etc. However, I'm struggling to decide whether this is in fact the best way, or, is one-hot encoding more suitable.

The reason is because I don't think I'm trying to rank the candidates by suitability, I'm merely trying to predict whether they will leave their current job, so it doesn't really make sense to me to say that a phd is better than any other level of education. It seems more like an 'arbitrary' partition in this context.

Am I right in thinking that one-hot encoding is better in a situation like this? Or am I simply overthinking and should just go with the naive definition?

EDIT: I read the answer to this question: What is an example of how ordinal data can yield bad classification results in contrast with one-hot encoding?

and have tried to apply it to my problem.

So by using ordinal encoding, I would be enforcing the relationship

$$\hat{y}_{phd} > \hat{y}_{mas} > \hat{y}_{grad} > \hat{y}_{high} > \hat{y}_{prim}$$.

Since I have a classification problem, I'm modelling probabilities, so the implication is that I'm enforcing the constraint that phd's have a higher probability of leaving their job without knowing whether that is true or not. Therefore, I need to use one-hot encoding.

Is this logic sound?

Source Link
ryan132442
  • 381
  • 2
  • 4

Advice for how to approach encoding non-numerical data

I have a dataset where the task is to classify whether someone looking for a new job will leave their current job, based on a number of factors. (dataset: https://www.kaggle.com/arashnic/hr-analytics-job-change-of-data-scientists)

I'm trying to encode the data and one of the things I struggle with is how to decide whether data is categorical, ordinal.

For example, one of the columns is called 'education_level', and has 5 levels, 'phd', 'masters', 'graduate', 'high school', 'primary school'.

Now if I were to encode this naively, then I'd say it was an ordinal variable because a phd is usually better than a masters, etc. However, I'm struggling to decide whether this is in fact the best way, or, is one-hot encoding more suitable.

The reason is because I don't think I'm trying to rank the candidates by suitability, I'm merely trying to predict whether they will leave their current job, so it doesn't really make sense to me to say that a phd is better than any other level of education. It seems more like an 'arbitrary' partition in this context.

Am I right in thinking that one-hot encoding is better in a situation like this? Or am I simply overthinking and should just go with the naive definition?