Skip to main content

Questions tagged [many-categories]

Categorical variables with large number of levels, and statistical methods for working with such variables (example: fused lasso).

Filter by
Sorted by
Tagged with
0 votes
1 answer
44 views

How do I quantify the effect of a factor with many levels?

I want to look at the effect of brood ID on fledging success (a binary variable) in a sample of wild birds. Brood ID has ~150 levels. I have performed a likelihood ratio test comparing two logistic ...
Emadeel's user avatar
0 votes
0 answers
15 views

Sample size requirements for evaluating a many-class classifier

I am working on defining the requirements for a test/evaluation dataset for a many-class classifier (n_classes ~ 1,000), and am working to address concerns about ensuring the statistical power of the ...
jf2qm's user avatar
  • 1
0 votes
0 answers
39 views

Incorporating group charasteristics in multi-output regression setting

I'm working on a multi-output regression problem involving the prediction of over 80 numerical targets using an equivalent number of numerical features. I have achieved encouraging results with ...
redamal's user avatar
0 votes
0 answers
28 views

Practically and probabilistically modeling discrete data that can take on a countable (potentially infinite) number of values?

As an example, let's say that we have observations of the price of some good at various points in time and would like to predict the distribution of the price of the good one point in the future from ...
QMath's user avatar
  • 451
0 votes
0 answers
20 views

discovering latent values, with extremely high cardinality categorical features

I think i know what I need to do here, but I want a gut check, and i might need some direction on specific packages and processing to use. My goal is to discover the latent value of products that ...
jrubins's user avatar
  • 13
1 vote
0 answers
57 views

Clustering or factor analysis for dimensionality reduction in multivariate linear regression

I have dataset describing aggregated purchases from multiple brands. It contains variables: Brand (ordinal) Promotion (ordinal) Sales (numeric) I need to use linear regression to describe the effect ...
Lazy Artist SQuex's user avatar
1 vote
3 answers
114 views

How best to regularize high-cardinality fixed effects?

Let's say that I have data in the 10s-100s of millions of observations. This data is clustered across hundreds, thousands, or even millions of entities (in a B2B context, these might be corporate ...
StatStudent19's user avatar
8 votes
1 answer
959 views

How to manage categorical variable with MANY categories

I have a dataset with some variables with MANY categories (one has about 20000, other about 2000, the third about 200). Need to make a multi-class prediction (not binary, but 3 values) How can I ...
Kaikus's user avatar
  • 219
0 votes
0 answers
175 views

Bias towards categorical data when one-hot encoding and standardizing (for machine learning)

I have a dataset containing a fair amount of continuous and categorical variables. I one-hot encode these variables to be used in various machine learning algorithms. Let's presume a variable has n ...
bob_cart's user avatar
0 votes
0 answers
63 views

Descision trees and splits on categorical variables with many levels? [duplicate]

This exercise document ask the following question (page 25): Your assistant, A, builds a decision tree to investigate which variables have a significant impact on response time. The variable day, ...
shawn's user avatar
  • 21
1 vote
1 answer
228 views

How to encode categorical variable with multiple categories per datapoint?

Consider this question on a survey: What desserts have you eaten? Apple pie Banana pudding Coconut cake Doughnut holes The user can pick as many of the options as they like. How would one encode ...
xojfqa's user avatar
  • 177
0 votes
0 answers
57 views

A categorical regressor with many categories: my linear regression coefficients do not make sense

Model: ...
Asia's user avatar
  • 1
1 vote
1 answer
48 views

Statistical test for count data in three different environments

I'm an undergrad student working on a small research project while studying abroad and I was looking for a statistical test that I could use to compare species richness between three forest types. I ...
Kian 's user avatar
  • 11
1 vote
1 answer
66 views

Plotting ideas for large number of unique catagories

I have 209 unique categories within a column, is there a nice way to plot it ? I am grouping the 209 unique categories with their respective cost column. How should I represent it visually ? Below ...
FalloutATS21's user avatar
2 votes
0 answers
166 views

Directed Acyclic Graph including a categorical variable with 20 levels

Is it possible within causal inference using DAGs to sensibly include a categorical variable with 20 levels? I have seen that regression trees can be used in this situation but not in combination with ...
ReadBeard's user avatar
  • 333
2 votes
1 answer
129 views

Classification algorithms for categorical predictors with extremely high cardinality

I am kinda confused with the above data. I have categorical predictors with so many unique groups, for example Treatment code variable has 15000+ unique codes, and Drug code variable has 800 unique ...
ForestGump's user avatar
2 votes
2 answers
346 views

Without encoding, how can we solve high cardinality issue?

I already referred the posts here but this question is different. I don't wish to use categorical encoding. details given below I have a dataset of 3000 unique customers purchase data. The dataset ...
The Great's user avatar
  • 3,302
3 votes
2 answers
742 views

Does it make sense to include ZIP code as a covariate in regression model?

Background I have a dataset representing a large group of people that I'm using to specify a Cox proportional hazards model of a binary outcome on some explanatory variables. My outcome variable is a ...
logjammin's user avatar
  • 729
1 vote
1 answer
609 views

Categorical variable with too many categories. Should I group them according to frequency or according to the target?

I am working with a dataset of flight records and I model the flight delay. I have variables for the origin and destination airport , but each of them has about 300 categories. I think about grouping ...
Manveru's user avatar
  • 177
0 votes
1 answer
1k views

Rule of thumb for collapsing categorical variables with many levels?

First of all, this question is related to this one: Principled way of collapsing categorical variables with many levels? but I think the scope of the answers I'm looking for is different. Just to ...
danton's user avatar
  • 1
0 votes
2 answers
184 views

Can I do Bayesian Logistic Regression of multiple categorical variables one by one?

My main background knowledge about Bayesian analysis comes from Doing Bayesian Data Analysis by John K. Kruschke. I have a dataset with observations y (success, fail) and several categorical variables ...
M. Chen's user avatar
  • 103
0 votes
0 answers
52 views

Unsupervised clustering with a categorical with tens of thousands of levels

I need to perform a clustering analysis of a medical claims dataset to identify anomalous healthcare providers. My dataset contains a variable called diagnosis code ...
TheGoat's user avatar
  • 639
0 votes
1 answer
678 views

Combining levels of a categorial variable in an ordinal logit regression?

Categorical variables in the ordered logit model. Can I combine levels of variables like income together? If yes, what are the key factors to consider in deciding how to combine levels?
Samin Ba's user avatar
0 votes
1 answer
172 views

Highly important categorical variable with many values and only few data points per value

Let's say I've got a dataset of music albums. As predictors, I have the artist, the genre, the year it was made plus several others (categorical and numeric). I want to predict the number of copies ...
Kolti's user avatar
  • 1
1 vote
0 answers
99 views

Unable to get good performance from my dataset having high cardinality

I have a multiclass classification problem. In the dataset, I have five categorical variables each having 1730, 235, 60,20 and 5 unique categories in each respectively. Apart from that I have 4 ...
Shubham Kaushik's user avatar
1 vote
1 answer
91 views

What statistical test its appropriate for my experimental design?

i need some help with my research. I dont know if its possible to use a statistic test in my design. To exemplify, following the design: 1 - I fertilize and distribute 100 embryos in each well (W1, W2,...
Leanderson Silva's user avatar
1 vote
0 answers
107 views

Random effect just because of many levels

I have seen a suggestion that if there are a large number of levels of a factor, one ought to treat them as random effects. I think it has come up in several places, but most recently I read it in The ...
Vallo Varik's user avatar
2 votes
1 answer
79 views

Dealing with text column of thousands different values

I have this dataset with some numerical and some text columns and want to create an ML forecasting model. The thing is that one column called 'diagnosis' is text (each entry is one sentence long) and ...
hippocampus's user avatar
0 votes
0 answers
152 views

Multiple categorical variables transform to dummy

I'm developing a linear regression model that contains multiple categorical explanatory variables (e.g., cities, marital status), including other binary and continous variables. The output is 0/1 ...
visu_hello's user avatar
1 vote
0 answers
35 views

Encode categorical variables with many labels

I am trying to predict a multiclass categorical outcome variable by comparing different classifier algorithms. I've got a dataset that includes two categorical variables that have many labels (>...
LeLuc's user avatar
  • 661
0 votes
0 answers
133 views

Options when model complexity and separation causes non-convergence in logistic regression

I have created an example data set here My data represent the presence/absence of a particular animal species (data$outcome) and measurements of trees. I would like ...
Pat Taggart's user avatar
1 vote
1 answer
572 views

How to handle the dummy variables with overlapping categories?

Background of The Question Let's say, I have four categories (A, B, C, D). Considering one (D) as a reference variable, there will be three categories on which I have to work. But the problem is one ...
Md Sabbir Ahmed's user avatar
4 votes
4 answers
4k views

How to visualise data where one variable is continuous and the other is categorical?

This question is very simple but I have been struggling in getting the right script for this. My data set goes as follows: ...
Sofia's user avatar
  • 41
1 vote
2 answers
109 views

Performing regression on a dataset with lots of categories

I am trying to work on a price prediction model, the attributes have lots of categories and all these categories are coded as integers. I am assuming if I build a regression model on this, the model ...
Karthik K V's user avatar
0 votes
1 answer
15 views

spatially explicit longitudinal categorical change response to predictor variables

I'm doing a land use/land cover change (LULCC) analysis with annual data spanning 10 years. The land cover class pixels change annually (ie A -> B -> D -> A). There are 5 nominal response ...
Britt Smith's user avatar
1 vote
2 answers
2k views

Target Encoder for Logistic regression

I have a data set that has many categorical values, I want to build a linear model using Logistic regression algorithm. One way of handling Categorical variables is ...
deltascience's user avatar
4 votes
1 answer
1k views

Up to what number of distinct values should I transform a categorical variable in a dummy variable?

When working with categorical variables, it's common to do some sort of transformation. Usually people apply a one-hot encoding. Putting it simply, we transform a categorical into a dummy variable. ...
trder's user avatar
  • 660
2 votes
2 answers
4k views

Optimal binning methods for categorical variables

I'm running a multinomial logit to predict the outcome of a categoric response variable. I have both continuous and categoric independent variables, and I know it's bad practicde to bin the ...
amestrian's user avatar
  • 265
0 votes
1 answer
532 views

Preprocessing a data set for linear regression

I'm currently a student in a machine learning course studying for an upcoming exam. Here's a question I've been given for practice: You have a very large dataset of employees and you'd like to ...
user avatar
0 votes
1 answer
359 views

Higher Order Category Overlap Analysis

I am attempting to analyse the categorical overlap of a dataset to ultimately ascertain the optimal way of categorising the data to minimise the amount of used categories to describe the dataset. ...
iwinallS's user avatar
3 votes
1 answer
124 views

Does it have any meaning to compute the $\chi^2$ and the exact Fisher test on big contingency tables

I have several datasets containing integers. I want to perform a bivariate analysis between a specific subset of variables. However, some of them have a lot of modalities. Is computing a $\chi^2$ ...
Jérôme Richard's user avatar
0 votes
0 answers
103 views

Sampling Technique: Categorical Data, Many Levels

I have a data set that has a categorical variable with almost half the number of observations as categories. Certain categories have only one observation. A minimal reproducable example in R would ...
user2550228's user avatar
1 vote
0 answers
33 views

Converting Continuous variable to Categorical [duplicate]

When should one consider converting continuous variable into categorical variable ? Are there guidelines ? Is it justified to bin skewed variable ? How should I determine the range / binning when I do ...
learner's user avatar
  • 617
3 votes
1 answer
114 views

Does this violate the assumption of independence for regression

A very basic question which I have never encountered a discussion of before. I am conducting bivariate logistic regression (although my question applies to linear models as well). I have 11,500 ...
user54285's user avatar
  • 1,091
-1 votes
1 answer
521 views

too much levels in the categorical variable in a GLM

I have 187 observations, the categorical variable is a predictor. My response variable is CPUE (catch per unit of effort). My goal is to know which of these variables (temperature, chlorophyll, depth, ...
ina's user avatar
  • 1
1 vote
2 answers
7k views

Performing one-hot encoding on a very large dataset

I am currently analysis a data set containing 654281 observations and 27 variables. I aim to perform binary logistic regression and many of my variables are categorical. I know one hot encoding is ...
Sydney's user avatar
  • 51
2 votes
1 answer
671 views

How to productionize a k-fold target-encoded feature?

I am attempting to build a model that has many predictors which are both categorical and large in cardinality. Target encoding looks to be a good solution for including these features, but I'm unsure ...
Darrrrrren's user avatar
0 votes
0 answers
26 views

How to deal with 100+ levels in categorical variable in multiple linear regression? [duplicate]

Im trying to model: Y~x0+x1+x2+x3+x4, were Y is a continous variable (cost), x0 is the ...
Jam.Wil's user avatar
  • 77
1 vote
1 answer
185 views

Select top-k feature from a categorical variable using $\chi^2$

I am working with a categorical variable that has a lot of levels (let's say more than 20). I would like to binarize all the levels doing one-hot-encoding in order to use these new variables in a ...
A1010's user avatar
  • 213
0 votes
1 answer
516 views

Statistical significance of a relationship of two categorical fields with more than two classes

From my dataset, I have two columns called the cuisine and the restaurant-grade. Each column corresponds to a restaurant. There are 6 different cuisines and 5 different grades. The question that I am ...
realkes's user avatar
  • 141

1
2 3 4 5 6