# Questions tagged [many-categories]

Categorical variables with large number of levels, and statistical methods for working with such variables (example: fused lasso).

248 questions
Filter by
Sorted by
Tagged with
12 views

### Binning and WoE transformation. Reducing number of categories for high cardinality features

I'm doing a credit default risk project. I have some features like a job title that has >100000 unique titles. What is the best way to reduce cardinality in a meaningful way? The end goal is to get ...
14 views

### Binary logistic regression with dummy variables for several different IVs

I want to carry out a binary regression where the DV is 0 = Never considered giving up pet, 1 = have considered giving up pet. I have several categorical variables that I want to enter into the model: ...
76 views

### Does it make sense to include ZIP code as a covariate in regression model?

Background I have a dataset representing a large group of people that I'm using to specify a Cox proportional hazards model of a binary outcome on some explanatory variables. My outcome variable is a ...
• 455
1 vote
33 views

### Categorical variable with too many categories. Should I group them according to frequency or according to the target?

I am working with a dataset of flight records and I model the flight delay. I have variables for the origin and destination airport , but each of them has about 300 categories. I think about grouping ...
• 135
17 views

### Correlation between trinary categorical variable and continuous variable

I have a categorical variable that can take three values: -1, -2, or -3. Another variable is continuous. How can I quantify how well the categorical variable predicts the continuous one? I am finding ...
• 101
95 views

### Rule of thumb for collapsing categorical variables with many levels?

First of all, this question is related to this one: Principled way of collapsing categorical variables with many levels? but I think the scope of the answers I'm looking for is different. Just to ...
47 views

### Can I do Bayesian Logistic Regression of multiple categorical variables one by one?

My main background knowledge about Bayesian analysis comes from Doing Bayesian Data Analysis by John K. Kruschke. I have a dataset with observations y (success, fail) and several categorical variables ...
29 views

### Unsupervised clustering with a categorical with tens of thousands of levels

I need to perform a clustering analysis of a medical claims dataset to identify anomalous healthcare providers. My dataset contains a variable called diagnosis code ...
• 549
44 views

### How should i combine different levels of categorial variable?

I have a question about categorical variables in the ordered logit model. can I add different levels of variables like income together to have fewer levels of a variable? or if no, on which basis ...
36 views

### Highly important categorical variable with many values and only few data points per value

Let's say I've got a dataset of music albums. As predictors, I have the artist, the genre, the year it was made plus several others (categorical and numeric). I want to predict the number of copies ...
25 views

### Fitting a machine learning model to data set with numerical features and a categorical feature with large cardinality

I am seeking advice for a data set that I am working with as I am new to data science. Suppose that the features are $P, X_1, \dots, X_n$ and $Y$ is the response. For simplicity, I will treat $Y$ as a ...
• 165
1 vote
47 views

### Unable to get good performance from my dataset having high cardinality

I have a multiclass classification problem. In the dataset, I have five categorical variables each having 1730, 235, 60,20 and 5 unique categories in each respectively. Apart from that I have 4 ...
1 vote
51 views

### What statistical test its appropriate for my experimental design?

i need some help with my research. I dont know if its possible to use a statistic test in my design. To exemplify, following the design: 1 - I fertilize and distribute 100 embryos in each well (W1, W2,...
1 vote
30 views

### Random effect just because of many levels

I have seen a suggestion that if there are a large number of levels of a factor, one ought to treat them as random effects. I think it has come up in several places, but most recently I read it in The ...
20 views

### Sampling from multiple distributions per weights

I have multiple distributions, e.g., a variable may be sampled from a normal distribution $50\%$ of the time, and a uniform distribution $50\%$ of the time. This is simple enough to code, but is there ...
37 views

### Dealing with text column of thousands different values

I have this dataset with some numerical and some text columns and want to create an ML forecasting model. The thing is that one column called 'diagnosis' is text (each entry is one sentence long) and ...
10 views

### Any issues with conducting stratafied train/test splitting based on the distribution of a categorical predictor?

I am building a xgboost regressor for a dataset that includes a categorical feature with a very large number of levels (on average, each level has an observation frequency of only about .2%). However, ...
40 views

### How to use a categorical covariates with high dimensionality in survival analysis

I am performing survival analysis on a dataset which contains mostly numerical variables, and binary categorical variables. However there is only one categorical variable which has up to 20 different ...
• 360
35 views

### Multiple categorical variables transform to dummy

I'm developing a linear regression model that contains multiple categorical explanatory variables (e.g., cities, marital status), including other binary and continous variables. The output is 0/1 ...
1 vote
25 views

### Encode categorical variables with many labels

I am trying to predict a multiclass categorical outcome variable by comparing different classifier algorithms. I've got a dataset that includes two categorical variables that have many labels (>...
• 641
53 views

• 1,669
855 views

### Alternatives to using dummy variables?

I am working on this dataset: https://www.kaggle.com/russellyates88/suicide-rates-overview-1985-to-2016, and it has a lot of categorical variables, while I am more used to work with the continuous ...
• 511
184 views

### Using label encoder on a categorical feature that we want to embed

I have a dataset with feature that have very high cardinality, doing one-hot encoding is not an option because of memory limitations, so I am currently label encoding this feature and then I feed that ...
240 views

### Gradient boosting (GB) splitting methods (categorical features)

Regarding categorical features - ordinary trees treat categorical features in two main ways, CART - considers only binary splitting, those computing the mean response value (y_mean_i per each category ...
• 21
1 vote