Questions tagged [many-categories]
Categorical variables with large number of levels, and statistical methods for working with such variables (example: fused lasso).
263
questions
0
votes
0
answers
16
views
discovering latent values, with extremely high cardinality categorical features
I think i know what I need to do here, but I want a gut check, and i might need some direction on specific packages and processing to use.
My goal is to discover the latent value of products that ...
- 13
1
vote
0
answers
53
views
Clustering or factor analysis for dimensionality reduction in multivariate linear regression
I have dataset describing aggregated purchases from multiple brands. It contains variables:
Brand (ordinal)
Promotion (ordinal)
Sales (numeric)
I need to use linear regression to describe the effect ...
1
vote
3
answers
73
views
How best to regularize high-cardinality fixed effects?
Let's say that I have data in the 10s-100s of millions of observations. This data is clustered across hundreds, thousands, or even millions of entities (in a B2B context, these might be corporate ...
- 331
0
votes
0
answers
10
views
Testing difference of counts of one category across a combination of the other two categories
I am using R. I have a dataset that looks like this using srt():
...
- 111
0
votes
0
answers
9
views
Disbalanced categorical variable after grouping
I have a categorical variable with A LOT of categories (about 2000), in a set with 100000 observations. There are a few representative categories (may be 2000 to 1000 different values), but the rest ...
- 219
8
votes
1
answer
426
views
How to manage categorical variable with MANY categories
I have a dataset with some variables with MANY categories (one has about 20000, other about 2000, the third about 200). Need to make a multi-class prediction (not binary, but 3 values)
How can I ...
- 219
0
votes
0
answers
90
views
Bias towards categorical data when one-hot encoding and standardizing (for machine learning)
I have a dataset containing a fair amount of continuous and categorical variables. I one-hot encode these variables to be used in various machine learning algorithms.
Let's presume a variable has n ...
- 63
0
votes
0
answers
63
views
Descision trees and splits on categorical variables with many levels? [duplicate]
This exercise document ask the following question (page 25):
Your assistant, A, builds a decision tree to investigate which
variables have a significant impact on response time. The variable
day, ...
- 11
1
vote
1
answer
149
views
How to encode categorical variable with multiple categories per datapoint?
Consider this question on a survey:
What desserts have you eaten?
Apple pie
Banana pudding
Coconut cake
Doughnut holes
The user can pick as many of the options as they like. How would one encode ...
- 137
0
votes
0
answers
45
views
0
votes
0
answers
26
views
Correlation between a status and count data
Correlation between categorical and zero inflated count data
Context : A company have 5 production locations and each of the company can select their production status from any of the following :
...
- 1
1
vote
1
answer
41
views
Statistical test for count data in three different environments
I'm an undergrad student working on a small research project while studying abroad and I was looking for a statistical test that I could use to compare species richness between three forest types.
I ...
- 11
0
votes
0
answers
50
views
Model selection when all independent variables are booleans
I am trying to do some basic seasonal adjustments at a daily periodicity so that I can then do some high-frequency forecasting. The dependent variable for the training data is what % of events ...
1
vote
1
answer
50
views
Plotting ideas for large number of unique catagories
I have 209 unique categories within a column, is there a nice way to plot it ?
I am grouping the 209 unique categories with their respective cost column. How should I represent it visually ?
Below ...
- 23
2
votes
0
answers
120
views
Directed Acyclic Graph including a categorical variable with 20 levels
Is it possible within causal inference using DAGs to sensibly include a categorical variable with 20 levels? I have seen that regression trees can be used in this situation but not in combination with ...
- 333
0
votes
0
answers
26
views
Check significance of each particular value in a Categorical variable
Type of Fuel
Count of Fuel Type
CNG
6000
Diesel
3500
Petrol
3500
LPG
500
I want to check significance of each type of Fuel in this particular table. My end goal is to check whether I can club LPG ...
2
votes
1
answer
106
views
Classification algorithms for categorical predictors with extremely high cardinality
I am kinda confused with the above data. I have categorical predictors with so many unique groups, for example Treatment code variable has 15000+ unique codes, and Drug code variable has 800 unique ...
- 327
2
votes
2
answers
263
views
Without encoding, how can we solve high cardinality issue?
I already referred the posts here but this question is different. I don't wish to use categorical encoding. details given below
I have a dataset of 3000 unique customers purchase data. The dataset ...
- 2,894
2
votes
2
answers
509
views
Does it make sense to include ZIP code as a covariate in regression model?
Background
I have a dataset representing a large group of people that I'm using to specify a Cox proportional hazards model of a binary outcome on some explanatory variables. My outcome variable is a ...
- 689
1
vote
1
answer
421
views
Categorical variable with too many categories. Should I group them according to frequency or according to the target?
I am working with a dataset of flight records and I model the flight delay. I have variables for the origin and destination airport , but each of them has about 300 categories. I think about grouping ...
- 177
0
votes
1
answer
723
views
Rule of thumb for collapsing categorical variables with many levels?
First of all, this question is related to this one: Principled way of collapsing categorical variables with many levels? but I think the scope of the answers I'm looking for is different.
Just to ...
- 1
0
votes
2
answers
140
views
Can I do Bayesian Logistic Regression of multiple categorical variables one by one?
My main background knowledge about Bayesian analysis comes from Doing Bayesian Data Analysis by John K. Kruschke.
I have a dataset with observations y (success, fail) and several categorical variables ...
- 103
0
votes
0
answers
49
views
Unsupervised clustering with a categorical with tens of thousands of levels
I need to perform a clustering analysis of a medical claims dataset to identify anomalous healthcare providers.
My dataset contains a variable called diagnosis code ...
- 639
0
votes
1
answer
598
views
Combining levels of a categorial variable in an ordinal logit regression?
Categorical variables in the ordered logit model.
Can I combine levels of variables like income together? If yes, what are the key factors to consider in deciding how to combine levels?
- 1
0
votes
1
answer
140
views
Highly important categorical variable with many values and only few data points per value
Let's say I've got a dataset of music albums. As predictors, I have the artist, the genre, the year it was made plus several others (categorical and numeric). I want to predict the number of copies ...
- 1
1
vote
0
answers
88
views
Unable to get good performance from my dataset having high cardinality
I have a multiclass classification problem. In the dataset, I have five categorical variables each having 1730, 235, 60,20 and 5 unique categories in each respectively. Apart from that I have 4 ...
1
vote
1
answer
66
views
What statistical test its appropriate for my experimental design?
i need some help with my research. I dont know if its possible to use a statistic test in my design.
To exemplify, following the design:
1 - I fertilize and distribute 100 embryos in each well (W1, W2,...
1
vote
0
answers
67
views
Random effect just because of many levels
I have seen a suggestion that if there are a large number of levels of a factor, one ought to treat them as random effects. I think it has come up in several places, but most recently I read it in The ...
- 77
2
votes
1
answer
61
views
Dealing with text column of thousands different values
I have this dataset with some numerical and some text columns and want to create an ML forecasting model. The thing is that one column called 'diagnosis' is text (each entry is one sentence long) and ...
- 165
0
votes
0
answers
102
views
How to use a categorical covariates with high dimensionality in survival analysis
I am performing survival analysis on a dataset which contains mostly numerical variables, and binary categorical variables. However there is only one categorical variable which has up to 20 different ...
- 360
0
votes
0
answers
135
views
Multiple categorical variables transform to dummy
I'm developing a linear regression model that contains multiple categorical explanatory variables (e.g., cities, marital status), including other binary and continous variables.
The output is 0/1 ...
1
vote
0
answers
34
views
Encode categorical variables with many labels
I am trying to predict a multiclass categorical outcome variable by comparing different classifier algorithms.
I've got a dataset that includes two categorical variables that have many labels (>...
- 651
0
votes
0
answers
111
views
Options when model complexity and separation causes non-convergence in logistic regression
I have created an example data set here
My data represent the presence/absence of a particular animal species (data$outcome) and measurements of trees. I would like ...
- 829
1
vote
1
answer
517
views
How to handle the dummy variables with overlapping categories?
Background of The Question
Let's say, I have four categories (A, B, C, D). Considering one (D) as a reference variable, there will be three categories on which I have to work. But the problem is one ...
- 294
4
votes
4
answers
3k
views
How to visualise data where one variable is continuous and the other is categorical?
This question is very simple but I have been struggling in getting the right script for this.
My data set goes as follows:
...
- 41
0
votes
2
answers
99
views
Performing regression on a dataset with lots of categories
I am trying to work on a price prediction model, the attributes have lots of categories and all these categories are coded as integers. I am assuming if I build a regression model on this, the model ...
0
votes
1
answer
15
views
spatially explicit longitudinal categorical change response to predictor variables
I'm doing a land use/land cover change (LULCC) analysis with annual data spanning 10 years. The land cover class pixels change annually (ie A -> B -> D -> A). There are 5 nominal response ...
- 11
1
vote
2
answers
1k
views
Target Encoder for Logistic regression
I have a data set that has many categorical values, I want to build a linear model using Logistic regression algorithm.
One way of handling Categorical variables is ...
- 385
4
votes
1
answer
921
views
Up to what number of distinct values should I transform a categorical variable in a dummy variable?
When working with categorical variables, it's common to do some sort of transformation. Usually people apply a one-hot encoding. Putting it simply, we transform a categorical into a dummy variable. ...
- 640
2
votes
2
answers
3k
views
Optimal binning methods for categorical variables
I'm running a multinomial logit to predict the outcome of a categoric response variable.
I have both continuous and categoric independent variables, and I know it's bad practicde to bin the ...
- 255
0
votes
1
answer
504
views
Preprocessing a data set for linear regression
I'm currently a student in a machine learning course studying for an upcoming exam. Here's a question I've been given for practice:
You have a very large dataset of employees and you'd like to ...
user295785
0
votes
1
answer
312
views
Higher Order Category Overlap Analysis
I am attempting to analyse the categorical overlap of a dataset to ultimately ascertain the optimal way of categorising the data to minimise the amount of used categories to describe the dataset.
...
- 1
3
votes
1
answer
116
views
Does it have any meaning to compute the $\chi^2$ and the exact Fisher test on big contingency tables
I have several datasets containing integers. I want to perform a bivariate analysis between a specific subset of variables. However, some of them have a lot of modalities. Is computing a $\chi^2$ ...
0
votes
0
answers
99
views
Sampling Technique: Categorical Data, Many Levels
I have a data set that has a categorical variable with almost half the number of observations as categories.
Certain categories have only one observation.
A minimal reproducable example in R would ...
- 211
1
vote
0
answers
32
views
Converting Continuous variable to Categorical [duplicate]
When should one consider converting continuous variable into categorical variable ? Are there guidelines ? Is it justified to bin skewed variable ?
How should I determine the range / binning when I do ...
- 587
3
votes
1
answer
98
views
Does this violate the assumption of independence for regression
A very basic question which I have never encountered a discussion of before. I am conducting bivariate logistic regression (although my question applies to linear models as well). I have 11,500 ...
- 1,031
-1
votes
1
answer
382
views
too much levels in the categorical variable in a GLM
I have 187 observations, the categorical variable is a predictor. My response variable is CPUE (catch per unit of effort). My goal is to know which of these variables (temperature, chlorophyll, depth, ...
- 1
1
vote
2
answers
5k
views
Performing one-hot encoding on a very large dataset
I am currently analysis a data set containing 654281 observations and 27 variables. I aim to perform binary logistic regression and many of my variables are categorical.
I know one hot encoding is ...
- 51
1
vote
1
answer
497
views
How to productionize a k-fold target-encoded feature?
I am attempting to build a model that has many predictors which are both categorical and large in cardinality. Target encoding looks to be a good solution for including these features, but I'm unsure ...
- 153
0
votes
0
answers
26
views
How to deal with 100+ levels in categorical variable in multiple linear regression? [duplicate]
Im trying to model: Y~x0+x1+x2+x3+x4, were Y is a continous variable (cost), x0 is the ...
- 45