Questions tagged [many-categories]

Categorical variables with large number of levels, and statistical methods for working with such variables (example: fused lasso).

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

Binning and WoE transformation. Reducing number of categories for high cardinality features

I'm doing a credit default risk project. I have some features like a job title that has >100000 unique titles. What is the best way to reduce cardinality in a meaningful way? The end goal is to get ...
user avatar
0 votes
0 answers
14 views

Binary logistic regression with dummy variables for several different IVs

I want to carry out a binary regression where the DV is 0 = Never considered giving up pet, 1 = have considered giving up pet. I have several categorical variables that I want to enter into the model: ...
user avatar
2 votes
2 answers
76 views

Does it make sense to include ZIP code as a covariate in regression model?

Background I have a dataset representing a large group of people that I'm using to specify a Cox proportional hazards model of a binary outcome on some explanatory variables. My outcome variable is a ...
user avatar
  • 455
1 vote
1 answer
33 views

Categorical variable with too many categories. Should I group them according to frequency or according to the target?

I am working with a dataset of flight records and I model the flight delay. I have variables for the origin and destination airport , but each of them has about 300 categories. I think about grouping ...
user avatar
  • 135
0 votes
0 answers
17 views

Correlation between trinary categorical variable and continuous variable

I have a categorical variable that can take three values: -1, -2, or -3. Another variable is continuous. How can I quantify how well the categorical variable predicts the continuous one? I am finding ...
user avatar
  • 101
0 votes
1 answer
95 views

Rule of thumb for collapsing categorical variables with many levels?

First of all, this question is related to this one: Principled way of collapsing categorical variables with many levels? but I think the scope of the answers I'm looking for is different. Just to ...
user avatar
  • 1
0 votes
2 answers
47 views

Can I do Bayesian Logistic Regression of multiple categorical variables one by one?

My main background knowledge about Bayesian analysis comes from Doing Bayesian Data Analysis by John K. Kruschke. I have a dataset with observations y (success, fail) and several categorical variables ...
user avatar
0 votes
0 answers
29 views

Unsupervised clustering with a categorical with tens of thousands of levels

I need to perform a clustering analysis of a medical claims dataset to identify anomalous healthcare providers. My dataset contains a variable called diagnosis code ...
user avatar
  • 549
0 votes
1 answer
44 views

How should i combine different levels of categorial variable?

I have a question about categorical variables in the ordered logit model. can I add different levels of variables like income together to have fewer levels of a variable? or if no, on which basis ...
user avatar
0 votes
1 answer
36 views

Highly important categorical variable with many values and only few data points per value

Let's say I've got a dataset of music albums. As predictors, I have the artist, the genre, the year it was made plus several others (categorical and numeric). I want to predict the number of copies ...
user avatar
  • 1
0 votes
0 answers
25 views

Fitting a machine learning model to data set with numerical features and a categorical feature with large cardinality

I am seeking advice for a data set that I am working with as I am new to data science. Suppose that the features are $P, X_1, \dots, X_n$ and $Y$ is the response. For simplicity, I will treat $Y$ as a ...
user avatar
1 vote
0 answers
47 views

Unable to get good performance from my dataset having high cardinality

I have a multiclass classification problem. In the dataset, I have five categorical variables each having 1730, 235, 60,20 and 5 unique categories in each respectively. Apart from that I have 4 ...
user avatar
1 vote
1 answer
51 views

What statistical test its appropriate for my experimental design?

i need some help with my research. I dont know if its possible to use a statistic test in my design. To exemplify, following the design: 1 - I fertilize and distribute 100 embryos in each well (W1, W2,...
user avatar
1 vote
0 answers
30 views

Random effect just because of many levels

I have seen a suggestion that if there are a large number of levels of a factor, one ought to treat them as random effects. I think it has come up in several places, but most recently I read it in The ...
user avatar
0 votes
0 answers
20 views

Sampling from multiple distributions per weights

I have multiple distributions, e.g., a variable may be sampled from a normal distribution $50\%$ of the time, and a uniform distribution $50\%$ of the time. This is simple enough to code, but is there ...
user avatar
2 votes
1 answer
37 views

Dealing with text column of thousands different values

I have this dataset with some numerical and some text columns and want to create an ML forecasting model. The thing is that one column called 'diagnosis' is text (each entry is one sentence long) and ...
user avatar
0 votes
0 answers
10 views

Any issues with conducting stratafied train/test splitting based on the distribution of a categorical predictor?

I am building a xgboost regressor for a dataset that includes a categorical feature with a very large number of levels (on average, each level has an observation frequency of only about .2%). However, ...
user avatar
  • 1
0 votes
0 answers
40 views

How to use a categorical covariates with high dimensionality in survival analysis

I am performing survival analysis on a dataset which contains mostly numerical variables, and binary categorical variables. However there is only one categorical variable which has up to 20 different ...
user avatar
  • 360
0 votes
0 answers
35 views

Multiple categorical variables transform to dummy

I'm developing a linear regression model that contains multiple categorical explanatory variables (e.g., cities, marital status), including other binary and continous variables. The output is 0/1 ...
user avatar
1 vote
0 answers
25 views

Encode categorical variables with many labels

I am trying to predict a multiclass categorical outcome variable by comparing different classifier algorithms. I've got a dataset that includes two categorical variables that have many labels (>...
user avatar
  • 641
0 votes
0 answers
53 views

Options when model complexity and separation causes non-convergence in logistic regression

I have created an example data set here My data represent the presence/absence of a particular animal species (data$outcome) and measurements of trees. I would like ...
user avatar
0 votes
1 answer
175 views

How to handle the dummy variables with overlapping categories?

Background of The Question Let's say, I have four categories (A, B, C, D). Considering one (D) as a reference variable, there will be three categories on which I have to work. But the problem is one ...
user avatar
5 votes
4 answers
266 views

How to visualise data where one variable is continuous and the other is categorical?

This question is very simple but I have been struggling in getting the right script for this. My data set goes as follows: ...
user avatar
  • 51
0 votes
2 answers
72 views

Performing regression on a dataset with lots of categories

I am trying to work on a price prediction model, the attributes have lots of categories and all these categories are coded as integers. I am assuming if I build a regression model on this, the model ...
user avatar
0 votes
1 answer
13 views

spatially explicit longitudinal categorical change response to predictor variables

I'm doing a land use/land cover change (LULCC) analysis with annual data spanning 10 years. The land cover class pixels change annually (ie A -> B -> D -> A). There are 5 nominal response ...
user avatar
1 vote
2 answers
589 views

Target Encoder for Logistic regression

I have a data set that has many categorical values, I want to build a linear model using Logistic regression algorithm. One way of handling Categorical variables is ...
user avatar
4 votes
1 answer
629 views

Up to what number of distinct values should I transform a categorical variable in a dummy variable?

When working with categorical variables, it's common to do some sort of transformation. Usually people apply a one-hot encoding. Putting it simply, we transform a categorical into a dummy variable. ...
user avatar
  • 610
1 vote
2 answers
1k views

Optimal binning methods for categorical variables

I'm running a multinomial logit to predict the outcome of a categoric response variable. I have both continuous and categoric independent variables, and I know it's bad practicde to bin the ...
user avatar
  • 233
0 votes
1 answer
331 views

Preprocessing a data set for linear regression

I'm currently a student in a machine learning course studying for an upcoming exam. Here's a question I've been given for practice: You have a very large dataset of employees and you'd like to ...
user avatar
0 votes
1 answer
107 views

Higher Order Category Overlap Analysis

I am attempting to analyse the categorical overlap of a dataset to ultimately ascertain the optimal way of categorising the data to minimise the amount of used categories to describe the dataset. ...
user avatar
3 votes
1 answer
90 views

Does it have any meaning to compute the $\chi^2$ and the exact Fisher test on big contingency tables

I have several datasets containing integers. I want to perform a bivariate analysis between a specific subset of variables. However, some of them have a lot of modalities. Is computing a $\chi^2$ ...
user avatar
0 votes
0 answers
69 views

Sampling Technique: Categorical Data, Many Levels

I have a data set that has a categorical variable with almost half the number of observations as categories. Certain categories have only one observation. A minimal reproducable example in R would ...
user avatar
1 vote
0 answers
30 views

Converting Continuous variable to Categorical [duplicate]

When should one consider converting continuous variable into categorical variable ? Are there guidelines ? Is it justified to bin skewed variable ? How should I determine the range / binning when I do ...
user avatar
  • 547
3 votes
1 answer
51 views

Does this violate the assumption of independence for regression

A very basic question which I have never encountered a discussion of before. I am conducting bivariate logistic regression (although my question applies to linear models as well). I have 11,500 ...
user avatar
  • 881
-1 votes
1 answer
107 views

too much levels in the categorical variable in a GLM

I have 187 observations, the categorical variable is a predictor. My response variable is CPUE (catch per unit of effort). My goal is to know which of these variables (temperature, chlorophyll, depth, ...
user avatar
  • 1
1 vote
2 answers
2k views

Performing one-hot encoding on a very large dataset

I am currently analysis a data set containing 654281 observations and 27 variables. I aim to perform binary logistic regression and many of my variables are categorical. I know one hot encoding is ...
user avatar
  • 31
1 vote
1 answer
125 views

How to productionize a k-fold target-encoded feature?

I am attempting to build a model that has many predictors which are both categorical and large in cardinality. Target encoding looks to be a good solution for including these features, but I'm unsure ...
user avatar
0 votes
0 answers
26 views

How to deal with 100+ levels in categorical variable in multiple linear regression? [duplicate]

Im trying to model: Y~x0+x1+x2+x3+x4, were Y is a continous variable (cost), x0 is the ...
user avatar
  • 45
1 vote
1 answer
83 views

Select top-k feature from a categorical variable using $\chi^2$

I am working with a categorical variable that has a lot of levels (let's say more than 20). I would like to binarize all the levels doing one-hot-encoding in order to use these new variables in a ...
user avatar
  • 193
0 votes
1 answer
68 views

Statistical significance of a relationship of two categorical fields with more than two classes

From my dataset, I have two columns called the cuisine and the restaurant-grade. Each column corresponds to a restaurant. There are 6 different cuisines and 5 different grades. The question that I am ...
user avatar
  • 141
0 votes
1 answer
275 views

Dealing with over 1000 categorical values (which are also a unique identifiers)

I am preparing my dataset for a logistic regression and need to check how best to handle a column with categorical values. As the dataset is for sales transactions, the column in question is the ...
user avatar
1 vote
0 answers
207 views

Handling rare levels in a categorical variable? (or maybe it's not categorical at all)

I have a dataset where I'm trying to predict completion time of an application. There are a number of numeric and categorical predictors, with a one group of predictors being holds. An application may ...
user avatar
  • 41
1 vote
1 answer
61 views

Statistical measures for variable selection?

I have a data set which has multiple features (26) of high cardinality (categorical), low cardinality (categorical), numerical type. I wanted to select features for the target (numerical) prediction. ...
user avatar
  • 31
1 vote
0 answers
31 views

Amount of information a row adds to a data set

I'm sorry if this is vaguely worded, but I'm looking for a way to score each row in a dataset by, essentially, the amount of information that that row adds - or the uniqueness of that row in the data ...
user avatar
  • 31
0 votes
0 answers
35 views

Estimating coefficients of a large categorical variable

I'm trying to fit a GLM model with a categorical variable with 400 categories, and I'd like to reduce the number of categories. There are some categories with a lot of data, and a lot of categories ...
user avatar
1 vote
1 answer
512 views

What happens when you merge dummy variables together?

Suppose I want to regress $X$ on $y$, controlling for categorical $z$ with $100$ different levels. I believe that linear regression is appropriate. Normally I would create dummies $D_i$ for each ...
user avatar
  • 1,669
4 votes
1 answer
855 views

Alternatives to using dummy variables?

I am working on this dataset: https://www.kaggle.com/russellyates88/suicide-rates-overview-1985-to-2016, and it has a lot of categorical variables, while I am more used to work with the continuous ...
user avatar
  • 511
0 votes
0 answers
184 views

Using label encoder on a categorical feature that we want to embed

I have a dataset with feature that have very high cardinality, doing one-hot encoding is not an option because of memory limitations, so I am currently label encoding this feature and then I feed that ...
user avatar
2 votes
0 answers
240 views

Gradient boosting (GB) splitting methods (categorical features)

Regarding categorical features - ordinary trees treat categorical features in two main ways, CART - considers only binary splitting, those computing the mean response value (y_mean_i per each category ...
user avatar
  • 21
1 vote
2 answers
1k views

How to correctly account for country effects in logistic regression?

I use a database with entries at firm-level in 12 countries in 2008. I try to estimate innovation (0/1) based on few firm-level variables. I also want to see if / how much innovation is also due to ...
user avatar
  • 21

1
2 3 4 5