Linked Questions

58
votes
6answers
13k views

Principled way of collapsing categorical variables with many levels?

What techniques are available for collapsing (or pooling) many categories to a few, for the purpose of using them as an input (predictor) in a statistical model? Consider a variable like college ...
2
votes
4answers
5k views

Linear regression including categorical variables with hundreds of levels [duplicate]

I am trying to teach myself data science by solving some of the problems available on the internet. Currently I am trying to predict a fraud event with the aid of 4 categorical variables. Each of the ...
2
votes
1answer
847 views

Grouping predictor factor levels based on response variable [duplicate]

I've read that it's bad to do this, but am looking for details as to why. Suppose we're trying to fit the linear model $Y_i = \beta_0 + \beta_1 X_{1i} + \beta_2 X_{2i} + \epsilon_i$ where $Y$ is ...
2
votes
1answer
661 views

categorical data Analysis with too many levels [duplicate]

I am running multinomial logistic regression Model for rating Hotels. I have a variable called CHAIN ID which is a bunch of numbers but there are too many such IDs,...
2
votes
2answers
147 views

Reduce feature levels [duplicate]

I would like to know if someone knows of a way to group the number of levels of a feature that has 100's (even 1000's) of levels to a smaller number of levels - also, what number levels it should ...
2
votes
0answers
76 views

Clustering, reducing number of levels of categorical variable [duplicate]

I'm dealing with this big dataset which has: 1 categorical variable with 90 levels that represent some sort of "geographical area" 3 continuous variables What I'm trying to do is to "aggregate" the ...
1
vote
0answers
56 views

How to deal with categorical features with thousands of levels? [duplicate]

I have a query that I perform, I have a dataset that has several categorical features with thousands of levels. Applying get_dummies would generate a dataset that I could not work with. I would be ...
10
votes
3answers
11k views

R package for combining factor levels for datamining?

Wondering if anyone has run across a package/function in R that will combine levels of a factor whose proportion of all the levels in a factor is less than some threshold? Specifically, one of the ...
14
votes
2answers
4k views

Methods for merging / reducing categories in ordinal or nominal data?

I'm struggling to find a method for reducing the number of categories in nominal or ordinal data. For example, let's say that I want to build a regression model on a dataset that has a number of ...
12
votes
2answers
13k views

How to handle categorical predictors with too many levels? [duplicate]

I think it may be a problem if we directly use dummy variable for a categorical predictor having hundreds of levels. I have found one solution from the book 'Element of Statistical Learning'. The ...
1
vote
3answers
5k views

How to deal with categorical features in machine learning models?

For instance I have a set of categories from a variable called 'category'. From python, here are my categories and their corresponding counts. ...
2
votes
4answers
3k views

RandomForest factor with too many levels

I have a model with about 200,000 training observations, where I am regressing, with 4 factors and 2 continuous variables. One of my features has 927 levels, which is causing the R implementation of ...
5
votes
1answer
2k views

Pooling levels of categorical variables for regression trees

I have a data set I would like to do a regression analysis for. There are many features of both categorical and continuous types. One of the categorical features has many (>75) levels so this is an ...
2
votes
2answers
1k views

What's the model to use with too many levels for categorical predictors of a data set with 5 million rows in R?

I am trying to predict the traffic volume for all the stations. My dataset consists of two response variables (double), the total traffic volume and the net flow traffic volume. There are 5 ...
9
votes
1answer
742 views

Penalized methods for categorical data: combining levels in a factor

Penalized models can be used to estimate models where the number of parameters is equal to or even greater than the sample size. This situation can arise in log-linear models of large sparse tables of ...

15 30 50 per page