Converting nominal variables and ordinal variables in dataframe - categorical variables

Question

I have a data frame in which I have some categorical variables, some are ordinal and others are nominal.

How can I deal with nominal columns/variables that have too many levels? For example, I have a column native_country which would be a nominal variable.

Is there a way of converting such columns like the one mentioned above the way we do it with ordinal variables? Meaning that, when I am converting for ordinal columns you can have only one column for all levels each one having its unique number. But for nominal variables, different methods generate one column per level.

I am working with python and skit-learn library, so for now I decided to use OrdinalEncoder for ordinal columns, and OneHotEncoder for nominal columns.

The link to the dataset: https://www.kaggle.com/datasets/uciml/adult-census-income

Answer 1

Besides the options listed by @dx2-66 in their answer, it may be quite useful to use related external data if you can, depending on the problem you're trying to solve.

For example, if you want to convert native_country to a quantitative variable, you could use the GDP per capita of this country, instead of its name. Or if you want to stick to categorical variables for some reason, you could convert the country name to the World Bank category it belongs to -it will reduce drastically the number of categories you have.

There is a boatload of indicators you could consider using, see https://en.wikipedia.org/wiki/List_of_international_rankings for a non-exhaustive list. Beware, many international ranking indicators are associated to the GDP, so confounding is a risk to consider when you choose some indicator(s) to replace your categorical variable. Choosing the right indicator(s) may be a matter of domain expertise, not just a purely statistical problem.

Answer 2

So basically, your're dealing with unordered categories of high cardinality. Is some information loss acceptable?

If yes, you may use feature hashing or bitstrings. Otherwise leave one out variety of mean target encoding is the default choice. There are other options, like weight of evidence, but you should be cautious about the target leakage.

Most of the advanced encoding algorithms are implemented by the category_encoders library (nicely compatible with sklearn).

There's also a nice encoding method selection flowchart with several info links.