Questions tagged [many-categories]

Categorical variables with large number of levels, and statistical methods for working with such variables (example: fused lasso).

Filter by
Sorted by
Tagged with
85
votes
6answers
26k views

Principled way of collapsing categorical variables with many levels?

What techniques are available for collapsing (or pooling) many categories to a few, for the purpose of using them as an input (predictor) in a statistical model? Consider a variable like college ...
41
votes
6answers
81k views

Improve classification with many categorical variables

I'm working on a dataset with 200,000+ samples and approximately 50 features per sample: 10 continuous variables and the other ~40 are categorical variables (countries, languages, scientific fields ...
31
votes
8answers
23k views

how to represent geography or zip code in machine learning model or recommender system?

I am building a model and I think that geographic location is likely to be very good at predicting my target variable. I have the zip code of each of my users. I am not entirely sure about the best ...
29
votes
6answers
11k views

Problems with pie charts

There seems to be in increasing discussion about pie charts. The main arguments against it seem to be: Area is perceived with less power than length. Pie charts have very low data-point-to-pixel ...
22
votes
5answers
47k views

R's randomForest can not handle more than 32 levels. What is workaround?

R's randomForest package can not handle factor with more than 32 levels. When it is given more than 32 levels, it emits an error message: Can not handle categorical predictors with more than 32 ...
17
votes
3answers
10k views

Problems with one-hot encoding vs. dummy encoding

I am aware of the fact that categorical variables with k levels should be encoded with k-1 variables in dummy encoding (similarly for multi-valued categorical variables). I was wondering how much of a ...
16
votes
5answers
13k views

Fixed effect vs random effect when all possibilities are included in a mixed effects model

In a mixed effects model the recommendation is to use a fixed effect to estimate a parameter if all possible levels are included (e.g., both males and females). It is further recommended to use a ...
16
votes
2answers
10k views

Encoding categorical features to numbers for machine learning

Many machine learning algorithms, for example neural networks, expect to deal with numbers. So, when you have a categorical data, you need to convert it. By categorical I mean, for example: Car ...
15
votes
2answers
5k views

Methods for merging / reducing categories in ordinal or nominal data?

I'm struggling to find a method for reducing the number of categories in nominal or ordinal data. For example, let's say that I want to build a regression model on a dataset that has a number of ...
14
votes
2answers
15k views

How to handle categorical predictors with too many levels? [duplicate]

I think it may be a problem if we directly use dummy variable for a categorical predictor having hundreds of levels. I have found one solution from the book 'Elements of Statistical Learning' (p.329)...
13
votes
2answers
18k views

Will decision trees perform splitting of nodes by converting categorical values to numerical in practice?

In Decision Trees, when doing classification or regression, do we use only numerical values? Suppose I have a categorical column Wind as a feature. Suppose I am ...
11
votes
1answer
4k views

Reducing number of levels of unordered categorical predictor variable

I want to train a classifier, say SVM, or random forest, or any other classifier. One of the features in the dataset is a categorical variable with 1000 levels. What is the best way to reduce the ...
10
votes
3answers
12k views

R package for combining factor levels for datamining?

Wondering if anyone has run across a package/function in R that will combine levels of a factor whose proportion of all the levels in a factor is less than some threshold? Specifically, one of the ...
10
votes
1answer
941 views

Penalized methods for categorical data: combining levels in a factor

Penalized models can be used to estimate models where the number of parameters is equal to or even greater than the sample size. This situation can arise in log-linear models of large sparse tables of ...
9
votes
1answer
12k views

Encoding of categorical variables with high cardinality

For unsupervised anomaly detection / fraud analytics on credit card data (where I don't have labeled fraudulent cases), there are a lot of variables to consider. The data is of mixed type with ...
9
votes
2answers
18k views

Preprocess categorical variables with many values [duplicate]

I have a dataset that consists of only categorical variables and a target variable. I want to predict the (binary) target variable with the categorical variables. I am trying to do this in Python and ...
8
votes
1answer
2k views

Where to find a guide to encoding categorical features?

I am facing an ML task with various categorical variables. Some examples include the following: Binary variables (0,1). Multilevel factors that can be ordered (low, medium, high). Multilevel factors ...
8
votes
1answer
5k views

Encoding high-cardinality (many-category) categorical features when features greatly differ on the cardinality

I have been looking through questions regarding categorical feature encoding, but couldn't find any which discuss my problem. Apologies if I missed it. Let's say we have a dataset with binary and ...
8
votes
1answer
1k views

Dealing with Postcode in Regression?

I'm building a logistic regression model and one of the variables I have is postcode, I might be over thinking this but is it fine for me to leave postcode as is and regress it as: ...
7
votes
3answers
894 views

Can categorical data only take finitely or countably infinitely many values?

I wonder if categorical data by definition can only take finitely or countably infinitely many values? And no more i.e. not uncountably many values? Related question: is the distribution of a ...
7
votes
1answer
21k views

New factors levels not present in training data

I am getting the "New factors levels not present in training data" error. But I checked the nlevels and class for every column in development as well as test data and they are the same. Any plausible ...
7
votes
1answer
3k views

Pooling levels of categorical variables for regression trees

I have a data set I would like to do a regression analysis for. There are many features of both categorical and continuous types. One of the categorical features has many (>75) levels so this is an ...
6
votes
1answer
237 views

How to build a predictive model when more levels of a categorical predictor are possible than appear in the training data

I have created an Artificial Neural Network with 4 categorical features and a binary outcome either 1 for suspicious or 0 for non-suspicious: ...
6
votes
1answer
2k views

Getting rid of a huge categorical factor in multiple regression

I have a large regression problem with a lot of cases, but relatively few independent variables. One of them is a categorical factor with thousands of levels. Robust regression runs forever. In some ...
6
votes
1answer
3k views

How to visualize both total counts of categories and proportions of subcategories in a plot?

Suppose I have samples drawn from categories A, B, C. Within those categories, I have subcategories d,e,f which are found in all 3 categories. I want to visualize how many samples I have form ...
6
votes
1answer
3k views

How to compute the AUROC for a single categorical variable

I am building new features for a binary classifier. The new features fall into two categories: categorical and ordinal. An example of the first feature would be the colours ...
6
votes
1answer
715 views

Multi-label classification: Predict product category

I want to predict to which product category a product belongs. A total of 400k products need to be translated from the old (less refined) to the new product category tree. (E.g. alarm clock used to ...
6
votes
0answers
1k views

Categorical logit Predictor with too many different levels [duplicate]

One of the predictors I had in a logit model is "City". Problem is this categorical variable has too many factor levels. e.g. In a Sample of $\sim 3000$ there are already $\sim 200$ different cities. ...
5
votes
3answers
3k views

Is there any limitation for the number of categories in Multinomial logistic regression?

It know it's very general question but I'm wondering what kind of issues should I expect if the number of categories in my dependent variable (or even in predictor variables) are more than for ...
5
votes
2answers
3k views

Data visualization of average and standard deviation over a small time series

I am trying to find the best way to visualize the following data: I have values for 3 different times/dates, each time/date has the same 20 species. For each species I have the average height and ...
5
votes
1answer
1k views

The name of 'Fused' Lasso

As many of you know, the Fused Lasso is one of well known penalized methods, which is introduced by Tibshirani, 2005. However, I don't get to the meaning of how it is called. Could anyone give any ...
5
votes
4answers
108 views

How to visualise data where one variable is continuous and the other is categorical?

This question is very simple but I have been struggling in getting the right script for this. My data set goes as follows: ...
5
votes
2answers
247 views

In what cases is it OK to use categorical predictors with many levels in regression?

If $n\gg p$ ($n$ is the number of observations, $p$ is the number of dimensions), is it always OK to use categorical predictors with many levels in regression? Here $p$ is also pretty high as the ...
5
votes
2answers
334 views

How to work a data set with a huge number of features?

I'm new to machine learning so I apologize in advance if the answer is obvious. I have a dataset that includes an address feature. I was hoping to use one hot encoding to create a feature for each ...
5
votes
2answers
3k views

Categorical variable with a very large number of categories as a predictor

I am trying to use a categorical variable as a predictor in a supervised learning setting, but there are too many categories for the classification algorithm to handle, something like over a 1000 ...
5
votes
1answer
1k views

Deep Learning with many categories

Do deep learning algorithms run into trouble when tasked with classifying high dimensional input into one of many categories? By many I mean thousands or millions. If it does, how could one deal with ...
5
votes
1answer
2k views

How to fit OLS with many categorical levels, on more than one category

This question is not meant to be a software question, but I will illustrate the issue using R a bit. My Understanding of the Simple Case If I have a simple linear model with a categorical variable ...
5
votes
2answers
7k views

Too many dummy variables

If I'm doing a regression analysis and in my data I want to use quite a few categorical variables (for example region, educational level and political party they'd vote for), is a dummy variable ...
5
votes
1answer
2k views

Decision tree : handle attribute with many nominal values

I would like to build a decision tree from a training data. I have an attribute with many nominal values. For example, the department name attribute has about 20-30 values. I would like to group ...
4
votes
4answers
7k views

Linear regression including categorical variables with hundreds of levels [duplicate]

I am trying to teach myself data science by solving some of the problems available on the internet. Currently I am trying to predict a fraud event with the aid of 4 categorical variables. Each of the ...
4
votes
1answer
512 views

Effective data visualizations for large numbers of classes

Experimenting with a few approaches to visualize error modes/rates for models with a large number (100-500) of classes. For smaller numbers of classes have been using ROC graphs (with each class ...
4
votes
1answer
97 views

When it is useful to use dummy variables?

Is it is useful to use dummy variables even though it implies a big increase in the number of parameters? In my case, I have a training set of about 13000 observations, and I am using 6 parameters. ...
4
votes
1answer
7k views

How to handle too many categorical features with too many categories for XGBoost?

In my data I have 35 features and 14 of them are categorical. Half of them have 3 to 4 categories but others have 14 to 28 categories. One Hot Encoding them would only lead to a sparse matrix with ...
4
votes
1answer
887 views

Absent categorical data levels in Bootstrap samples

I have a huge dataset ($n$ around five million, $p$ around three thousand) for a classification problem, where my interest is predictive class probabilities for test data, not the target. I shall be ...
4
votes
2answers
278 views

Machine learning for multi-level response

I have a dataset with ~90000 observations and less than 10 features (all continuous). The problem is that the response variable has ~300 categories. Currently I would try to fit a multinomial linear ...
4
votes
1answer
617 views

Alternatives to using dummy variables?

I am working on this dataset: https://www.kaggle.com/russellyates88/suicide-rates-overview-1985-to-2016, and it has a lot of categorical variables, while I am more used to work with the continuous ...
4
votes
1answer
1k views

Encoding IP Address as a Predictor in Machine Learning

Is there some approach to "encoding" IP Address (IPv4) in a way that the new representation can capture both cardinality and the statistical distribution of the full range of IP address and also ...
4
votes
1answer
229 views

Maximum Number of Categorical Outcomes for Multinomial Logistic Regression?

Based on 4 or 5 predictor variables, I would like to determine "affinity" to a group of 26 different non-profits that an individual could potentially donate money to. I have approximately 1 million ...
4
votes
1answer
822 views

One hot encoding on a categorical variable with many values following a power-law distribution for use in logistic regression

I have a categorical variable (e.g., office locations) with about 500 values. The frequency of the values follow a power-law distribution (if you sort the categorical values by frequency descendingly),...
4
votes
1answer
7k views

Categorical variables with too many levels in machine learning [duplicate]

I have a machine learning problem where the dependent variable is binomial (Yes/No) and some of the independent variables are categorical (with more than 100 levels). I'm not sure whether dummy coding ...

1
2 3 4 5