Questions tagged [many-categories]

Categorical variables with large number of levels, and statistical methods for working with such variables (example: fused lasso).

Filter by
Sorted by
Tagged with
85
votes
6answers
26k views

Principled way of collapsing categorical variables with many levels?

What techniques are available for collapsing (or pooling) many categories to a few, for the purpose of using them as an input (predictor) in a statistical model? Consider a variable like college ...
9
votes
2answers
18k views

Preprocess categorical variables with many values [duplicate]

I have a dataset that consists of only categorical variables and a target variable. I want to predict the (binary) target variable with the categorical variables. I am trying to do this in Python and ...
16
votes
5answers
13k views

Fixed effect vs random effect when all possibilities are included in a mixed effects model

In a mixed effects model the recommendation is to use a fixed effect to estimate a parameter if all possible levels are included (e.g., both males and females). It is further recommended to use a ...
3
votes
3answers
2k views

Random Forest Regression with sparse data in Python

I am working on a Random Forest regression model to predict housing prices. I have about 500k rows of data with the following information: 1.House area in square meters. 2.Number of rooms. 3.City. ...
29
votes
6answers
11k views

Problems with pie charts

There seems to be in increasing discussion about pie charts. The main arguments against it seem to be: Area is perceived with less power than length. Pie charts have very low data-point-to-pixel ...
1
vote
1answer
2k views

One-hot-encoding gives untractable amount of classes

I'm performing regression on the price of bycicles based on their brand, model and submodel. These features are hierarchical: one model belongs only to one brand but one brand can have many models. ...
8
votes
1answer
2k views

Where to find a guide to encoding categorical features?

I am facing an ML task with various categorical variables. Some examples include the following: Binary variables (0,1). Multilevel factors that can be ordered (low, medium, high). Multilevel factors ...
13
votes
2answers
18k views

Will decision trees perform splitting of nodes by converting categorical values to numerical in practice?

In Decision Trees, when doing classification or regression, do we use only numerical values? Suppose I have a categorical column Wind as a feature. Suppose I am ...
22
votes
5answers
47k views

R's randomForest can not handle more than 32 levels. What is workaround?

R's randomForest package can not handle factor with more than 32 levels. When it is given more than 32 levels, it emits an error message: Can not handle categorical predictors with more than 32 ...
31
votes
8answers
23k views

how to represent geography or zip code in machine learning model or recommender system?

I am building a model and I think that geographic location is likely to be very good at predicting my target variable. I have the zip code of each of my users. I am not entirely sure about the best ...
7
votes
1answer
3k views

Pooling levels of categorical variables for regression trees

I have a data set I would like to do a regression analysis for. There are many features of both categorical and continuous types. One of the categorical features has many (>75) levels so this is an ...
10
votes
1answer
941 views

Penalized methods for categorical data: combining levels in a factor

Penalized models can be used to estimate models where the number of parameters is equal to or even greater than the sample size. This situation can arise in log-linear models of large sparse tables of ...
14
votes
2answers
15k views

How to handle categorical predictors with too many levels? [duplicate]

I think it may be a problem if we directly use dummy variable for a categorical predictor having hundreds of levels. I have found one solution from the book 'Elements of Statistical Learning' (p.329)...
17
votes
3answers
10k views

Problems with one-hot encoding vs. dummy encoding

I am aware of the fact that categorical variables with k levels should be encoded with k-1 variables in dummy encoding (similarly for multi-valued categorical variables). I was wondering how much of a ...
16
votes
2answers
10k views

Encoding categorical features to numbers for machine learning

Many machine learning algorithms, for example neural networks, expect to deal with numbers. So, when you have a categorical data, you need to convert it. By categorical I mean, for example: Car ...
9
votes
1answer
12k views

Encoding of categorical variables with high cardinality

For unsupervised anomaly detection / fraud analytics on credit card data (where I don't have labeled fraudulent cases), there are a lot of variables to consider. The data is of mixed type with ...
5
votes
1answer
2k views

Decision tree : handle attribute with many nominal values

I would like to build a decision tree from a training data. I have an attribute with many nominal values. For example, the department name attribute has about 20-30 values. I would like to group ...
6
votes
1answer
237 views

How to build a predictive model when more levels of a categorical predictor are possible than appear in the training data

I have created an Artificial Neural Network with 4 categorical features and a binary outcome either 1 for suspicious or 0 for non-suspicious: ...
5
votes
1answer
1k views

The name of 'Fused' Lasso

As many of you know, the Fused Lasso is one of well known penalized methods, which is introduced by Tibshirani, 2005. However, I don't get to the meaning of how it is called. Could anyone give any ...
41
votes
6answers
81k views

Improve classification with many categorical variables

I'm working on a dataset with 200,000+ samples and approximately 50 features per sample: 10 continuous variables and the other ~40 are categorical variables (countries, languages, scientific fields ...
6
votes
0answers
1k views

Categorical logit Predictor with too many different levels [duplicate]

One of the predictors I had in a logit model is "City". Problem is this categorical variable has too many factor levels. e.g. In a Sample of $\sim 3000$ there are already $\sim 200$ different cities. ...
8
votes
1answer
5k views

Encoding high-cardinality (many-category) categorical features when features greatly differ on the cardinality

I have been looking through questions regarding categorical feature encoding, but couldn't find any which discuss my problem. Apologies if I missed it. Let's say we have a dataset with binary and ...
5
votes
1answer
1k views

Deep Learning with many categories

Do deep learning algorithms run into trouble when tasked with classifying high dimensional input into one of many categories? By many I mean thousands or millions. If it does, how could one deal with ...
4
votes
1answer
887 views

Absent categorical data levels in Bootstrap samples

I have a huge dataset ($n$ around five million, $p$ around three thousand) for a classification problem, where my interest is predictive class probabilities for test data, not the target. I shall be ...
3
votes
1answer
997 views

Specifying a linear mixed model in lmer with replications nested within a fully crossed design

I’m trying to specify a linear mixed model for a somewhat complicated, nested & crossed method comparison study with replicated measurements. The goal is to partition and compare variances. It’s ...
3
votes
2answers
689 views

Label encoding for high-cardinality features in boosted decision trees - what to do with unseen labels?

I have a categorical feature with very high-cardinality (on the order of 1000s of unique IDs). RIght now, I am using label encoding along with XGBoost, because from what I understand, decision trees ...
3
votes
0answers
131 views

treatment for factors with many levels [duplicate]

I'm running a predictive model and I have one possible predictive variable that is a factor and has more than 800 levels. I tried to reduce it running ctree in R (with the variable as the only ...
3
votes
1answer
475 views

Categorical variable with a lot of levels and interaction

Suppose we have a categorical variable $X_1$ with $100$ levels. Should we not test interaction between $X$ and some other variable $X_2$? Because if, for example, $X_1(3)*X_2$ is not significant ...
2
votes
1answer
991 views

Grouping predictor factor levels based on response variable [duplicate]

I've read that it's bad to do this, but am looking for details as to why. Suppose we're trying to fit the linear model $Y_i = \beta_0 + \beta_1 X_{1i} + \beta_2 X_{2i} + \epsilon_i$ where $Y$ is ...
2
votes
0answers
151 views

Handling datasets with categorical variables of many levels [duplicate]

I am working on the Diabetes in 130 US hospitals for years 1999--2008 dataset. After removing unnecessary variables (i.e. some IDs or near-zero-variance variables) and doing some naive imptuation, I ...
2
votes
0answers
511 views

Fitting multilevel categorical variables with neural nets

Most of the neural net algorithms I'm aware of require multilevel, ANOVA-type categorical features to be preprocessed into a set of dummy (0,1) variables. So, if one has a single categorical feature ...
1
vote
1answer
56 views

Statistical measures for variable selection?

I have a data set which has multiple features (26) of high cardinality (categorical), low cardinality (categorical), numerical type. I wanted to select features for the target (numerical) prediction. ...
7
votes
3answers
894 views

Can categorical data only take finitely or countably infinitely many values?

I wonder if categorical data by definition can only take finitely or countably infinitely many values? And no more i.e. not uncountably many values? Related question: is the distribution of a ...
3
votes
1answer
73 views

Does it have any meaning to compute the $\chi^2$ and the exact Fisher test on big contingency tables

I have several datasets containing integers. I want to perform a bivariate analysis between a specific subset of variables. However, some of them have a lot of modalities. Is computing a $\chi^2$ ...
2
votes
0answers
99 views

Test of Independence Between Two Nominal Variables With Many Levels Each

I am looking to test whether there is a significant relationship between two nominal variables, one which has 100 levels and the other with 10 levels. I initially considered doing a $\chi^2$ test of ...
2
votes
0answers
112 views

What is the general procedure or general rules for grouping factor levels? [duplicate]

I am attempting to build a predictive (machine-learning) logistic regression model that contains mostly categorical (non-ordinal) variables. As part of a variable selection process I run a Pearson ...
2
votes
1answer
1k views

Encoding of categorical data/feature/predictor for binary classification

ML newbie here, currently looking at a binary classification problem. I have quite a good number of training data (easily over 50k) which consists of both numeric and categorical data. The categorical ...
1
vote
2answers
388 views

Performing one-hot encoding on a very large dataset

I am currently analysis a data set containing 654281 observations and 27 variables. I aim to perform binary logistic regression and many of my variables are categorical. I know one hot encoding is ...
1
vote
1answer
791 views

How to process categorical features with many values? [duplicate]

I want to apply machine learning and deep learning. I have categorical data on string. My first option was to perform dummy encoding on the columns (scikitlearn). ...
1
vote
1answer
572 views

Linear Regression and High Dimensional Categorical Data

I've read that mean encoding is useful for classification tasks with high dimensional categorical data. My question: What kinds of encodings are effective for high dimensional categorical data in ...
0
votes
1answer
33 views

Statistical significance of a relationship of two categorical fields with more than two classes

From my dataset, I have two columns called the cuisine and the restaurant-grade. Each column corresponds to a restaurant. There are 6 different cuisines and 5 different grades. The question that I am ...
0
votes
1answer
190 views

Logistic regression with categorical predictors

I'm trying to play around with classification models and started off with logistic regression in R. When I have all the numeric variables in the data set the model works correctly and I was able to ...