Questions tagged [many-categories]

Categorical variables with large number of levels, and statistical methods for working with such variables (example: fused lasso).

Filter by
Sorted by
Tagged with
107 votes
6 answers
41k views

Principled way of collapsing categorical variables with many levels?

What techniques are available for collapsing (or pooling) many categories to a few, for the purpose of using them as an input (predictor) in a statistical model? Consider a variable like college ...
shadowtalker's user avatar
  • 12.5k
44 votes
6 answers
92k views

Improve classification with many categorical variables

I'm working on a dataset with 200,000+ samples and approximately 50 features per sample: 10 continuous variables and the other ~40 are categorical variables (countries, languages, scientific fields ...
Bertrand R's user avatar
40 votes
7 answers
36k views

how to represent geography or zip code in machine learning model or recommender system?

I am building a model and I think that geographic location is likely to be very good at predicting my target variable. I have the zip code of each of my users. I am not entirely sure about the best ...
captain_ahab's user avatar
  • 1,512
32 votes
6 answers
13k views

Problems with pie charts

There seems to be in increasing discussion about pie charts. The main arguments against it seem to be: Area is perceived with less power than length. Pie charts have very low data-point-to-pixel ...
deps_stats's user avatar
  • 1,685
23 votes
5 answers
52k views

R's randomForest can not handle more than 32 levels. What is workaround?

R's randomForest package can not handle factor with more than 32 levels. When it is given more than 32 levels, it emits an error message: Can not handle categorical predictors with more than 32 ...
Minkoo Seo's user avatar
  • 1,260
21 votes
4 answers
32k views

Encoding of categorical variables with high cardinality

For unsupervised anomaly detection / fraud analytics on credit card data (where I don't have labeled fraudulent cases), there are a lot of variables to consider. The data is of mixed type with ...
robot_2077198's user avatar
19 votes
5 answers
18k views

Fixed effect vs random effect when all possibilities are included in a mixed effects model

In a mixed effects model the recommendation is to use a fixed effect to estimate a parameter if all possible levels are included (e.g., both males and females). It is further recommended to use a ...
gung - Reinstate Monica's user avatar
17 votes
3 answers
14k views

Problems with one-hot encoding vs. dummy encoding

I am aware of the fact that categorical variables with k levels should be encoded with k-1 variables in dummy encoding (similarly for multi-valued categorical variables). I was wondering how much of a ...
user90772's user avatar
  • 701
16 votes
2 answers
11k views

Encoding categorical features to numbers for machine learning

Many machine learning algorithms, for example neural networks, expect to deal with numbers. So, when you have a categorical data, you need to convert it. By categorical I mean, for example: Car ...
Nucular's user avatar
  • 453
15 votes
2 answers
6k views

Methods for merging / reducing categories in ordinal or nominal data?

I'm struggling to find a method for reducing the number of categories in nominal or ordinal data. For example, let's say that I want to build a regression model on a dataset that has a number of ...
Figaro's user avatar
  • 1,152
15 votes
2 answers
16k views

How to handle categorical predictors with too many levels? [duplicate]

I think it may be a problem if we directly use dummy variable for a categorical predictor having hundreds of levels. I have found one solution from the book 'Elements of Statistical Learning' (p.329)...
Jerry's user avatar
  • 378
13 votes
2 answers
19k views

Will decision trees perform splitting of nodes by converting categorical values to numerical in practice?

In Decision Trees, when doing classification or regression, do we use only numerical values? Suppose I have a categorical column Wind as a feature. Suppose I am ...
Sarath R Nair's user avatar
12 votes
1 answer
4k views

Reducing number of levels of unordered categorical predictor variable

I want to train a classifier, say SVM, or random forest, or any other classifier. One of the features in the dataset is a categorical variable with 1000 levels. What is the best way to reduce the ...
sabunime's user avatar
  • 121
11 votes
2 answers
22k views

Preprocess categorical variables with many values [duplicate]

I have a dataset that consists of only categorical variables and a target variable. I want to predict the (binary) target variable with the categorical variables. I am trying to do this in Python and ...
Bert Carremans's user avatar
10 votes
3 answers
12k views

R package for combining factor levels for datamining?

Wondering if anyone has run across a package/function in R that will combine levels of a factor whose proportion of all the levels in a factor is less than some threshold? Specifically, one of the ...
B_Miner's user avatar
  • 8,610
10 votes
1 answer
7k views

Encoding high-cardinality (many-category) categorical features when features greatly differ on the cardinality

I have been looking through questions regarding categorical feature encoding, but couldn't find any which discuss my problem. Apologies if I missed it. Let's say we have a dataset with binary and ...
user5226582's user avatar
10 votes
1 answer
1k views

Penalized methods for categorical data: combining levels in a factor

Penalized models can be used to estimate models where the number of parameters is equal to or even greater than the sample size. This situation can arise in log-linear models of large sparse tables of ...
andrewH's user avatar
  • 3,117
9 votes
2 answers
27k views

New factors levels not present in training data

I am getting the "New factors levels not present in training data" error. But I checked the nlevels and class for every column in development as well as test data and they are the same. Any plausible ...
Alex's user avatar
  • 91
8 votes
1 answer
2k views

Where to find a guide to encoding categorical features?

I am facing an ML task with various categorical variables. Some examples include the following: Binary variables (0,1). Multilevel factors that can be ordered (low, medium, high). Multilevel factors ...
mickkk's user avatar
  • 939
8 votes
3 answers
1k views

Can categorical data only take finitely or countably infinitely many values?

I wonder if categorical data by definition can only take finitely or countably infinitely many values? And no more i.e. not uncountably many values? Related question: is the distribution of a ...
Tim's user avatar
  • 19.4k
8 votes
1 answer
792 views

How to manage categorical variable with MANY categories

I have a dataset with some variables with MANY categories (one has about 20000, other about 2000, the third about 200). Need to make a multi-class prediction (not binary, but 3 values) How can I ...
Kaikus's user avatar
  • 219
8 votes
1 answer
2k views

Dealing with Postcode in Regression?

I'm building a logistic regression model and one of the variables I have is postcode, I might be over thinking this but is it fine for me to leave postcode as is and regress it as: ...
Grant McKinnon's user avatar
8 votes
1 answer
4k views

Pooling levels of categorical variables for regression trees

I have a data set I would like to do a regression analysis for. There are many features of both categorical and continuous types. One of the categorical features has many (>75) levels so this is an ...
Keith's user avatar
  • 521
7 votes
1 answer
4k views

How to visualize both total counts of categories and proportions of subcategories in a plot?

Suppose I have samples drawn from categories A, B, C. Within those categories, I have subcategories d,e,f which are found in all 3 categories. I want to visualize how many samples I have form ...
user4733's user avatar
  • 2,714
7 votes
1 answer
834 views

Multi-label classification: Predict product category

I want to predict to which product category a product belongs. A total of 400k products need to be translated from the old (less refined) to the new product category tree. (E.g. alarm clock used to ...
JobVisser21's user avatar
6 votes
1 answer
386 views

How to build a predictive model when more levels of a categorical predictor are possible than appear in the training data

I have created an Artificial Neural Network with 4 categorical features and a binary outcome either 1 for suspicious or 0 for non-suspicious: ...
sectechguy's user avatar
6 votes
1 answer
13k views

How to handle too many categorical features with too many categories for XGBoost?

In my data I have 35 features and 14 of them are categorical. Half of them have 3 to 4 categories but others have 14 to 28 categories. One Hot Encoding them would only lead to a sparse matrix with ...
Minila S's user avatar
6 votes
1 answer
2k views

Getting rid of a huge categorical factor in multiple regression

I have a large regression problem with a lot of cases, but relatively few independent variables. One of them is a categorical factor with thousands of levels. Robust regression runs forever. In some ...
Herbert's user avatar
  • 135
6 votes
1 answer
3k views

How to compute the AUROC for a single categorical variable

I am building new features for a binary classifier. The new features fall into two categories: categorical and ordinal. An example of the first feature would be the colours ...
Alex's user avatar
  • 4,382
6 votes
0 answers
1k views

Categorical logit Predictor with too many different levels [duplicate]

One of the predictors I had in a logit model is "City". Problem is this categorical variable has too many factor levels. e.g. In a Sample of $\sim 3000$ there are already $\sim 200$ different cities. ...
curious_cat's user avatar
  • 1,091
5 votes
3 answers
4k views

Is there any limitation for the number of categories in Multinomial logistic regression?

It know it's very general question but I'm wondering what kind of issues should I expect if the number of categories in my dependent variable (or even in predictor variables) are more than for ...
MFR's user avatar
  • 552
5 votes
2 answers
4k views

Data visualization of average and standard deviation over a small time series

I am trying to find the best way to visualize the following data: I have values for 3 different times/dates, each time/date has the same 20 species. For each species I have the average height and ...
Gago-Silva's user avatar
5 votes
1 answer
2k views

The name of 'Fused' Lasso

As many of you know, the Fused Lasso is one of well known penalized methods, which is introduced by Tibshirani, 2005. However, I don't get to the meaning of how it is called. Could anyone give any ...
inmybrain's user avatar
  • 538
5 votes
2 answers
282 views

In what cases is it OK to use categorical predictors with many levels in regression?

If $n\gg p$ ($n$ is the number of observations, $p$ is the number of dimensions), is it always OK to use categorical predictors with many levels in regression? Here $p$ is also pretty high as the ...
user98235's user avatar
  • 515
5 votes
2 answers
422 views

How to work a data set with a huge number of features?

I'm new to machine learning so I apologize in advance if the answer is obvious. I have a dataset that includes an address feature. I was hoping to use one hot encoding to create a feature for each ...
Abe Miessler's user avatar
5 votes
2 answers
3k views

Categorical variable with a very large number of categories as a predictor

I am trying to use a categorical variable as a predictor in a supervised learning setting, but there are too many categories for the classification algorithm to handle, something like over a 1000 ...
PT83's user avatar
  • 211
5 votes
1 answer
1k views

Deep Learning with many categories

Do deep learning algorithms run into trouble when tasked with classifying high dimensional input into one of many categories? By many I mean thousands or millions. If it does, how could one deal with ...
Taylor's user avatar
  • 20.6k
5 votes
1 answer
3k views

How to fit OLS with many categorical levels, on more than one category

This question is not meant to be a software question, but I will illustrate the issue using R a bit. My Understanding of the Simple Case If I have a simple linear model with a categorical variable ...
Corvus's user avatar
  • 5,345
5 votes
2 answers
8k views

Too many dummy variables

If I'm doing a regression analysis and in my data I want to use quite a few categorical variables (for example region, educational level and political party they'd vote for), is a dummy variable ...
Brian's user avatar
  • 303
5 votes
1 answer
2k views

Decision tree : handle attribute with many nominal values

I would like to build a decision tree from a training data. I have an attribute with many nominal values. For example, the department name attribute has about 20-30 values. I would like to group ...
wannik's user avatar
  • 235
5 votes
1 answer
10k views

Categorical variables with too many levels in machine learning [duplicate]

I have a machine learning problem where the dependent variable is binomial (Yes/No) and some of the independent variables are categorical (with more than 100 levels). I'm not sure whether dummy coding ...
Viswa's user avatar
  • 51
4 votes
4 answers
4k views

How to visualise data where one variable is continuous and the other is categorical?

This question is very simple but I have been struggling in getting the right script for this. My data set goes as follows: ...
Sofia's user avatar
  • 41
4 votes
4 answers
9k views

Linear regression including categorical variables with hundreds of levels [duplicate]

I am trying to teach myself data science by solving some of the problems available on the internet. Currently I am trying to predict a fraud event with the aid of 4 categorical variables. Each of the ...
Abhi's user avatar
  • 1,409
4 votes
4 answers
7k views

Regression with Lots of Categorical Variables

I'm facing a regression task with many categorical and few numeric features. I encoded them into dummies and removed the first dummy column for each feature. I am not getting very good R2 at all. I ...
Odisseo's user avatar
  • 674
4 votes
1 answer
1k views

Up to what number of distinct values should I transform a categorical variable in a dummy variable?

When working with categorical variables, it's common to do some sort of transformation. Usually people apply a one-hot encoding. Putting it simply, we transform a categorical into a dummy variable. ...
trder's user avatar
  • 650
4 votes
1 answer
1k views

Effective data visualizations for large numbers of classes

Experimenting with a few approaches to visualize error modes/rates for models with a large number (100-500) of classes. For smaller numbers of classes have been using ROC graphs (with each class ...
Slater Victoroff's user avatar
4 votes
1 answer
264 views

When it is useful to use dummy variables?

Is it is useful to use dummy variables even though it implies a big increase in the number of parameters? In my case, I have a training set of about 13000 observations, and I am using 6 parameters. ...
Simone's user avatar
  • 311
4 votes
1 answer
1k views

Absent categorical data levels in Bootstrap samples

I have a huge dataset ($n$ around five million, $p$ around three thousand) for a classification problem, where my interest is predictive class probabilities for test data, not the target. I shall be ...
Yoda's user avatar
  • 389
4 votes
1 answer
1k views

Why it might be bad to have too many feature levels

I am aware that a feature with too many levels might be bad for a number of algorithms (e.g. Logistic Regression). A typical approach to fix this would be to group the categories with a frequency ...
GRoutar's user avatar
  • 141
4 votes
2 answers
4k views

Categorical Data: Merging Categories?

I'm working with some categorical data that I want to use for prediction with a machine learning algorithm. Many of my my categorical features have multiple categories that contain very few positive ...
RDG's user avatar
  • 93

1
2 3 4 5 6