Questions tagged [many-categories]
Categorical variables with large number of levels, and statistical methods for working with such variables (example: fused lasso).
260
questions
107
votes
6
answers
41k
views
Principled way of collapsing categorical variables with many levels?
What techniques are available for collapsing (or pooling) many categories to a few, for the purpose of using them as an input (predictor) in a statistical model?
Consider a variable like college ...
44
votes
6
answers
92k
views
Improve classification with many categorical variables
I'm working on a dataset with 200,000+ samples and approximately 50 features per sample: 10 continuous variables and the other ~40 are categorical variables (countries, languages, scientific fields ...
40
votes
7
answers
36k
views
how to represent geography or zip code in machine learning model or recommender system?
I am building a model and I think that geographic location is likely to be very good at predicting my target variable. I have the zip code of each of my users. I am not entirely sure about the best ...
32
votes
6
answers
13k
views
Problems with pie charts
There seems to be in increasing discussion about pie charts.
The main arguments against it seem to be:
Area is perceived with less power than length.
Pie charts have very low data-point-to-pixel ...
23
votes
5
answers
52k
views
R's randomForest can not handle more than 32 levels. What is workaround?
R's randomForest package can not handle factor with more than 32 levels. When it is given more than 32 levels, it emits an error message:
Can not handle categorical predictors with more than 32 ...
21
votes
4
answers
32k
views
Encoding of categorical variables with high cardinality
For unsupervised anomaly detection / fraud analytics on credit card data (where I don't have labeled fraudulent cases), there are a lot of variables to consider. The data is of mixed type with ...
19
votes
5
answers
18k
views
Fixed effect vs random effect when all possibilities are included in a mixed effects model
In a mixed effects model the recommendation is to use a fixed effect to estimate a parameter if all possible levels are included (e.g., both males and females). It is further recommended to use a ...
17
votes
3
answers
14k
views
Problems with one-hot encoding vs. dummy encoding
I am aware of the fact that categorical variables with k levels should be encoded with k-1 variables in dummy encoding (similarly for multi-valued categorical variables). I was wondering how much of a ...
16
votes
2
answers
11k
views
Encoding categorical features to numbers for machine learning
Many machine learning algorithms, for example neural networks, expect to deal with numbers. So, when you have a categorical data, you need to convert it. By categorical I mean, for example:
Car ...
15
votes
2
answers
6k
views
Methods for merging / reducing categories in ordinal or nominal data?
I'm struggling to find a method for reducing the number of categories in nominal or ordinal data.
For example, let's say that I want to build a regression model on a dataset that has a number of ...
15
votes
2
answers
16k
views
How to handle categorical predictors with too many levels? [duplicate]
I think it may be a problem if we directly use dummy variable for a categorical predictor having hundreds of levels.
I have found one solution from the book 'Elements of Statistical Learning' (p.329)...
13
votes
2
answers
19k
views
Will decision trees perform splitting of nodes by converting categorical values to numerical in practice?
In Decision Trees, when doing classification or regression, do we use only numerical values? Suppose I have a categorical column Wind as a feature. Suppose I am ...
12
votes
1
answer
4k
views
Reducing number of levels of unordered categorical predictor variable
I want to train a classifier, say SVM, or random forest, or any other classifier. One of the features in the dataset is a categorical variable with 1000 levels. What is the best way to reduce the ...
11
votes
2
answers
22k
views
Preprocess categorical variables with many values [duplicate]
I have a dataset that consists of only categorical variables and a target variable. I want to predict the (binary) target variable with the categorical variables.
I am trying to do this in Python and ...
10
votes
3
answers
12k
views
R package for combining factor levels for datamining?
Wondering if anyone has run across a package/function in R that will combine levels of a factor whose proportion of all the levels in a factor is less than some threshold? Specifically, one of the ...
10
votes
1
answer
7k
views
Encoding high-cardinality (many-category) categorical features when features greatly differ on the cardinality
I have been looking through questions regarding categorical feature encoding, but couldn't find any which discuss my problem. Apologies if I missed it.
Let's say we have a dataset with binary and ...
10
votes
1
answer
1k
views
Penalized methods for categorical data: combining levels in a factor
Penalized models can be used to estimate models where the number of parameters is equal to or even greater than the sample size. This situation can arise in log-linear models of large sparse tables of ...
9
votes
2
answers
27k
views
New factors levels not present in training data
I am getting the "New factors levels not present in training data" error. But I checked the nlevels and class for every column in development as well as test data and they are the same. Any plausible ...
8
votes
1
answer
2k
views
Where to find a guide to encoding categorical features?
I am facing an ML task with various categorical variables. Some examples include the following:
Binary variables (0,1).
Multilevel factors that can be ordered (low, medium, high).
Multilevel factors ...
8
votes
3
answers
1k
views
Can categorical data only take finitely or countably infinitely many values?
I wonder if categorical data by definition can only take finitely or countably infinitely many values? And no more i.e. not uncountably many values?
Related question: is the distribution of a ...
8
votes
1
answer
792
views
How to manage categorical variable with MANY categories
I have a dataset with some variables with MANY categories (one has about 20000, other about 2000, the third about 200). Need to make a multi-class prediction (not binary, but 3 values)
How can I ...
8
votes
1
answer
2k
views
Dealing with Postcode in Regression?
I'm building a logistic regression model and one of the variables I have is postcode, I might be over thinking this but is it fine for me to leave postcode as is and regress it as:
...
8
votes
1
answer
4k
views
Pooling levels of categorical variables for regression trees
I have a data set I would like to do a regression analysis for. There are many features of both categorical and continuous types. One of the categorical features has many (>75) levels so this is an ...
7
votes
1
answer
4k
views
How to visualize both total counts of categories and proportions of subcategories in a plot?
Suppose I have samples drawn from categories A, B, C. Within those categories, I have subcategories d,e,f which are found in all 3 categories. I want to visualize how many samples I have form ...
7
votes
1
answer
834
views
Multi-label classification: Predict product category
I want to predict to which product category a product belongs. A total of 400k products need to be translated from the old (less refined) to the new product category tree. (E.g. alarm clock used to ...
6
votes
1
answer
386
views
How to build a predictive model when more levels of a categorical predictor are possible than appear in the training data
I have created an Artificial Neural Network with 4 categorical features and a binary outcome either 1 for suspicious or 0 for non-suspicious:
...
6
votes
1
answer
13k
views
How to handle too many categorical features with too many categories for XGBoost?
In my data I have 35 features and 14 of them are categorical. Half of them have 3 to 4 categories but others have 14 to 28 categories.
One Hot Encoding them would only lead to a sparse matrix with ...
6
votes
1
answer
2k
views
Getting rid of a huge categorical factor in multiple regression
I have a large regression problem with a lot of cases, but relatively few independent variables. One of them is a categorical factor with thousands of levels. Robust regression runs forever. In some ...
6
votes
1
answer
3k
views
How to compute the AUROC for a single categorical variable
I am building new features for a binary classifier. The new features fall into two categories: categorical and ordinal. An example of the first feature would be the colours ...
6
votes
0
answers
1k
views
Categorical logit Predictor with too many different levels [duplicate]
One of the predictors I had in a logit model is "City". Problem is this categorical variable has too many factor levels. e.g. In a Sample of $\sim 3000$ there are already $\sim 200$ different cities. ...
5
votes
3
answers
4k
views
Is there any limitation for the number of categories in Multinomial logistic regression?
It know it's very general question but I'm wondering what kind of issues should I expect if the number of categories in my dependent variable (or even in predictor variables) are more than for ...
5
votes
2
answers
4k
views
Data visualization of average and standard deviation over a small time series
I am trying to find the best way to visualize the following data:
I have values for 3 different times/dates, each time/date has the same 20 species. For each species I have the average height and ...
5
votes
1
answer
2k
views
The name of 'Fused' Lasso
As many of you know, the Fused Lasso is one of well known penalized methods, which is introduced by Tibshirani, 2005.
However, I don't get to the meaning of how it is called.
Could anyone give any ...
5
votes
2
answers
282
views
In what cases is it OK to use categorical predictors with many levels in regression?
If $n\gg p$ ($n$ is the number of observations, $p$ is the number of dimensions), is it always OK to use categorical predictors with many levels in regression? Here $p$ is also pretty high as the ...
5
votes
2
answers
422
views
How to work a data set with a huge number of features?
I'm new to machine learning so I apologize in advance if the answer is obvious.
I have a dataset that includes an address feature. I was hoping to use one hot encoding to create a feature for each ...
5
votes
2
answers
3k
views
Categorical variable with a very large number of categories as a predictor
I am trying to use a categorical variable as a predictor in a supervised learning setting, but there are too many categories for the classification algorithm to handle, something like over a 1000 ...
5
votes
1
answer
1k
views
Deep Learning with many categories
Do deep learning algorithms run into trouble when tasked with classifying high dimensional input into one of many categories? By many I mean thousands or millions. If it does, how could one deal with ...
5
votes
1
answer
3k
views
How to fit OLS with many categorical levels, on more than one category
This question is not meant to be a software question, but I will illustrate the issue using R a bit.
My Understanding of the Simple Case
If I have a simple linear model with a categorical variable ...
5
votes
2
answers
8k
views
Too many dummy variables
If I'm doing a regression analysis and in my data I want to use quite a few categorical variables (for example region, educational level and political party they'd vote for), is a dummy variable ...
5
votes
1
answer
2k
views
Decision tree : handle attribute with many nominal values
I would like to build a decision tree from a training data. I have an attribute with many nominal values. For example, the department name attribute has about 20-30 values. I would like to group ...
5
votes
1
answer
10k
views
Categorical variables with too many levels in machine learning [duplicate]
I have a machine learning problem where the dependent variable is binomial (Yes/No) and some of the independent variables are categorical (with more than 100 levels). I'm not sure whether dummy coding ...
4
votes
4
answers
4k
views
How to visualise data where one variable is continuous and the other is categorical?
This question is very simple but I have been struggling in getting the right script for this.
My data set goes as follows:
...
4
votes
4
answers
9k
views
Linear regression including categorical variables with hundreds of levels [duplicate]
I am trying to teach myself data science by solving some of the problems available on the internet.
Currently I am trying to predict a fraud event with the aid of 4 categorical variables. Each of the ...
4
votes
4
answers
7k
views
Regression with Lots of Categorical Variables
I'm facing a regression task with many categorical and few numeric features. I encoded them into dummies and removed the first dummy column for each feature.
I am not getting very good R2 at all. I ...
4
votes
1
answer
1k
views
Up to what number of distinct values should I transform a categorical variable in a dummy variable?
When working with categorical variables, it's common to do some sort of transformation. Usually people apply a one-hot encoding. Putting it simply, we transform a categorical into a dummy variable. ...
4
votes
1
answer
1k
views
Effective data visualizations for large numbers of classes
Experimenting with a few approaches to visualize error modes/rates for models with a large number (100-500) of classes. For smaller numbers of classes have been using ROC graphs (with each class ...
4
votes
1
answer
264
views
When it is useful to use dummy variables?
Is it is useful to use dummy variables even though it implies a big increase in the number of parameters?
In my case, I have a training set of about 13000 observations, and I am using 6 parameters. ...
4
votes
1
answer
1k
views
Absent categorical data levels in Bootstrap samples
I have a huge dataset ($n$ around five million, $p$ around three thousand) for a classification problem, where my interest is predictive class probabilities for test data, not the target. I shall be ...
4
votes
1
answer
1k
views
Why it might be bad to have too many feature levels
I am aware that a feature with too many levels might be bad for a number of algorithms (e.g. Logistic Regression).
A typical approach to fix this would be to group the categories with a frequency ...
4
votes
2
answers
4k
views
Categorical Data: Merging Categories?
I'm working with some categorical data that I want to use for prediction with a machine learning algorithm. Many of my my categorical features have multiple categories that contain very few positive ...