Questions tagged [many-categories]

Categorical variables with large number of levels, and statistical methods for working with such variables (example: fused lasso).

Filter by
Sorted by
Tagged with
1 vote
1 answer
31 views

Plotting ideas for large number of unique catagories

I have 209 unique categories within a column, is there a nice way to plot it ? I am grouping the 209 unique categories with their respective cost column. How should I represent it visually ? Below ...
0 votes
0 answers
15 views

Similarity between points with 3 categories

I have a large dataset with more than 1 million indexes. I need to find the most similar point. I cleared a little bit of dataset and left only these columns: "latitude"-float, "...
1 vote
0 answers
35 views

Directed Acyclic Graph including a categorical variable with 20 levels

Is it possible within causal inference using DAGs to sensibly include a categorical variable with 20 levels? I have seen that regression trees can be used in this situation but not in combination with ...
  • 167
0 votes
0 answers
18 views

Check significance of each particular value in a Categorical variable

Type of Fuel Count of Fuel Type CNG 6000 Diesel 3500 Petrol 3500 LPG 500 I want to check significance of each type of Fuel in this particular table. My end goal is to check whether I can club LPG ...
2 votes
1 answer
53 views

Classification algorithms for categorical predictors with extremely high cardinality

I am kinda confused with the above data. I have categorical predictors with so many unique groups, for example Treatment code variable has 15000+ unique codes, and Drug code variable has 800 unique ...
2 votes
2 answers
115 views

Without encoding, how can we solve high cardinality issue?

I already referred the posts here but this question is different. I don't wish to use categorical encoding. details given below I have a dataset of 3000 unique customers purchase data. The dataset ...
  • 2,096
0 votes
0 answers
58 views

Binning and WoE transformation. Reducing number of categories for high cardinality features

I'm doing a credit default risk project. I have some features like a job title that has >100000 unique titles. What is the best way to reduce cardinality in a meaningful way? The end goal is to get ...
0 votes
0 answers
38 views

Binary logistic regression with dummy variables for several different IVs

I want to carry out a binary regression where the DV is 0 = Never considered giving up pet, 1 = have considered giving up pet. I have several categorical variables that I want to enter into the model: ...
2 votes
2 answers
148 views

Does it make sense to include ZIP code as a covariate in regression model?

Background I have a dataset representing a large group of people that I'm using to specify a Cox proportional hazards model of a binary outcome on some explanatory variables. My outcome variable is a ...
  • 659
1 vote
1 answer
123 views

Categorical variable with too many categories. Should I group them according to frequency or according to the target?

I am working with a dataset of flight records and I model the flight delay. I have variables for the origin and destination airport , but each of them has about 300 categories. I think about grouping ...
  • 147
0 votes
1 answer
205 views

Rule of thumb for collapsing categorical variables with many levels?

First of all, this question is related to this one: Principled way of collapsing categorical variables with many levels? but I think the scope of the answers I'm looking for is different. Just to ...
  • 1
0 votes
2 answers
72 views

Can I do Bayesian Logistic Regression of multiple categorical variables one by one?

My main background knowledge about Bayesian analysis comes from Doing Bayesian Data Analysis by John K. Kruschke. I have a dataset with observations y (success, fail) and several categorical variables ...
0 votes
0 answers
37 views

Unsupervised clustering with a categorical with tens of thousands of levels

I need to perform a clustering analysis of a medical claims dataset to identify anomalous healthcare providers. My dataset contains a variable called diagnosis code ...
  • 549
0 votes
1 answer
185 views

How should i combine different levels of categorial variable?

I have a question about categorical variables in the ordered logit model. can I add different levels of variables like income together to have fewer levels of a variable? or if no, on which basis ...
0 votes
1 answer
62 views

Highly important categorical variable with many values and only few data points per value

Let's say I've got a dataset of music albums. As predictors, I have the artist, the genre, the year it was made plus several others (categorical and numeric). I want to predict the number of copies ...
  • 1
0 votes
0 answers
35 views

Fitting a machine learning model to data set with numerical features and a categorical feature with large cardinality

I am seeking advice for a data set that I am working with as I am new to data science. Suppose that the features are $P, X_1, \dots, X_n$ and $Y$ is the response. For simplicity, I will treat $Y$ as a ...
1 vote
0 answers
60 views

Unable to get good performance from my dataset having high cardinality

I have a multiclass classification problem. In the dataset, I have five categorical variables each having 1730, 235, 60,20 and 5 unique categories in each respectively. Apart from that I have 4 ...
1 vote
1 answer
53 views

What statistical test its appropriate for my experimental design?

i need some help with my research. I dont know if its possible to use a statistic test in my design. To exemplify, following the design: 1 - I fertilize and distribute 100 embryos in each well (W1, W2,...
1 vote
0 answers
32 views

Random effect just because of many levels

I have seen a suggestion that if there are a large number of levels of a factor, one ought to treat them as random effects. I think it has come up in several places, but most recently I read it in The ...
2 votes
1 answer
42 views

Dealing with text column of thousands different values

I have this dataset with some numerical and some text columns and want to create an ML forecasting model. The thing is that one column called 'diagnosis' is text (each entry is one sentence long) and ...
0 votes
0 answers
55 views

How to use a categorical covariates with high dimensionality in survival analysis

I am performing survival analysis on a dataset which contains mostly numerical variables, and binary categorical variables. However there is only one categorical variable which has up to 20 different ...
  • 350
0 votes
0 answers
63 views

Multiple categorical variables transform to dummy

I'm developing a linear regression model that contains multiple categorical explanatory variables (e.g., cities, marital status), including other binary and continous variables. The output is 0/1 ...
1 vote
0 answers
26 views

Encode categorical variables with many labels

I am trying to predict a multiclass categorical outcome variable by comparing different classifier algorithms. I've got a dataset that includes two categorical variables that have many labels (>...
  • 641
0 votes
0 answers
67 views

Options when model complexity and separation causes non-convergence in logistic regression

I have created an example data set here My data represent the presence/absence of a particular animal species (data$outcome) and measurements of trees. I would like ...
1 vote
1 answer
294 views

How to handle the dummy variables with overlapping categories?

Background of The Question Let's say, I have four categories (A, B, C, D). Considering one (D) as a reference variable, there will be three categories on which I have to work. But the problem is one ...
4 votes
4 answers
762 views

How to visualise data where one variable is continuous and the other is categorical?

This question is very simple but I have been struggling in getting the right script for this. My data set goes as follows: ...
  • 41
0 votes
2 answers
79 views

Performing regression on a dataset with lots of categories

I am trying to work on a price prediction model, the attributes have lots of categories and all these categories are coded as integers. I am assuming if I build a regression model on this, the model ...
0 votes
1 answer
14 views

spatially explicit longitudinal categorical change response to predictor variables

I'm doing a land use/land cover change (LULCC) analysis with annual data spanning 10 years. The land cover class pixels change annually (ie A -> B -> D -> A). There are 5 nominal response ...
1 vote
2 answers
837 views

Target Encoder for Logistic regression

I have a data set that has many categorical values, I want to build a linear model using Logistic regression algorithm. One way of handling Categorical variables is ...
4 votes
1 answer
676 views

Up to what number of distinct values should I transform a categorical variable in a dummy variable?

When working with categorical variables, it's common to do some sort of transformation. Usually people apply a one-hot encoding. Putting it simply, we transform a categorical into a dummy variable. ...
  • 610
1 vote
2 answers
2k views

Optimal binning methods for categorical variables

I'm running a multinomial logit to predict the outcome of a categoric response variable. I have both continuous and categoric independent variables, and I know it's bad practicde to bin the ...
  • 233
0 votes
1 answer
389 views

Preprocessing a data set for linear regression

I'm currently a student in a machine learning course studying for an upcoming exam. Here's a question I've been given for practice: You have a very large dataset of employees and you'd like to ...
's user avatar
0 votes
1 answer
133 views

Higher Order Category Overlap Analysis

I am attempting to analyse the categorical overlap of a dataset to ultimately ascertain the optimal way of categorising the data to minimise the amount of used categories to describe the dataset. ...
3 votes
1 answer
98 views

Does it have any meaning to compute the $\chi^2$ and the exact Fisher test on big contingency tables

I have several datasets containing integers. I want to perform a bivariate analysis between a specific subset of variables. However, some of them have a lot of modalities. Is computing a $\chi^2$ ...
0 votes
0 answers
72 views

Sampling Technique: Categorical Data, Many Levels

I have a data set that has a categorical variable with almost half the number of observations as categories. Certain categories have only one observation. A minimal reproducable example in R would ...
1 vote
0 answers
31 views

Converting Continuous variable to Categorical [duplicate]

When should one consider converting continuous variable into categorical variable ? Are there guidelines ? Is it justified to bin skewed variable ? How should I determine the range / binning when I do ...
  • 567
3 votes
1 answer
66 views

Does this violate the assumption of independence for regression

A very basic question which I have never encountered a discussion of before. I am conducting bivariate logistic regression (although my question applies to linear models as well). I have 11,500 ...
  • 1,021
-1 votes
1 answer
140 views

too much levels in the categorical variable in a GLM

I have 187 observations, the categorical variable is a predictor. My response variable is CPUE (catch per unit of effort). My goal is to know which of these variables (temperature, chlorophyll, depth, ...
  • 1
1 vote
2 answers
3k views

Performing one-hot encoding on a very large dataset

I am currently analysis a data set containing 654281 observations and 27 variables. I aim to perform binary logistic regression and many of my variables are categorical. I know one hot encoding is ...
  • 31
1 vote
1 answer
239 views

How to productionize a k-fold target-encoded feature?

I am attempting to build a model that has many predictors which are both categorical and large in cardinality. Target encoding looks to be a good solution for including these features, but I'm unsure ...
0 votes
0 answers
26 views

How to deal with 100+ levels in categorical variable in multiple linear regression? [duplicate]

Im trying to model: Y~x0+x1+x2+x3+x4, were Y is a continous variable (cost), x0 is the ...
  • 45
1 vote
1 answer
118 views

Select top-k feature from a categorical variable using $\chi^2$

I am working with a categorical variable that has a lot of levels (let's say more than 20). I would like to binarize all the levels doing one-hot-encoding in order to use these new variables in a ...
  • 193
0 votes
1 answer
141 views

Statistical significance of a relationship of two categorical fields with more than two classes

From my dataset, I have two columns called the cuisine and the restaurant-grade. Each column corresponds to a restaurant. There are 6 different cuisines and 5 different grades. The question that I am ...
  • 141
0 votes
1 answer
349 views

Dealing with over 1000 categorical values (which are also a unique identifiers)

I am preparing my dataset for a logistic regression and need to check how best to handle a column with categorical values. As the dataset is for sales transactions, the column in question is the ...
1 vote
0 answers
213 views

Handling rare levels in a categorical variable? (or maybe it's not categorical at all)

I have a dataset where I'm trying to predict completion time of an application. There are a number of numeric and categorical predictors, with a one group of predictors being holds. An application may ...
  • 41
1 vote
1 answer
62 views

Statistical measures for variable selection?

I have a data set which has multiple features (26) of high cardinality (categorical), low cardinality (categorical), numerical type. I wanted to select features for the target (numerical) prediction. ...
  • 31
1 vote
0 answers
31 views

Amount of information a row adds to a data set

I'm sorry if this is vaguely worded, but I'm looking for a way to score each row in a dataset by, essentially, the amount of information that that row adds - or the uniqueness of that row in the data ...
  • 31
0 votes
0 answers
35 views

Estimating coefficients of a large categorical variable

I'm trying to fit a GLM model with a categorical variable with 400 categories, and I'd like to reduce the number of categories. There are some categories with a lot of data, and a lot of categories ...
1 vote
1 answer
585 views

What happens when you merge dummy variables together?

Suppose I want to regress $X$ on $y$, controlling for categorical $z$ with $100$ different levels. I believe that linear regression is appropriate. Normally I would create dummies $D_i$ for each ...
  • 1,709
4 votes
1 answer
944 views

Alternatives to using dummy variables?

I am working on this dataset: https://www.kaggle.com/russellyates88/suicide-rates-overview-1985-to-2016, and it has a lot of categorical variables, while I am more used to work with the continuous ...
  • 531

1
2 3 4 5 6