Questions tagged [many-categories]

Categorical variables with large number of levels, and statistical methods for working with such variables (example: fused lasso).

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Any issues with conducting stratafied train/test splitting based on the distribution of a categorical predictor?

I am building a xgboost regressor for a dataset that includes a categorical feature with a very large number of levels (on average, each level has an observation frequency of only about .2%). However, ...
0
votes
0answers
11 views

How to use a categorical covariates with high dimensionality in survival analysis

I am performing survival analysis on a dataset which contains mostly numerical variables, and binary categorical variables. However there is only one categorical variable which has up to 20 different ...
0
votes
0answers
22 views

Multiple categorical variables transform to dummy

I'm developing a linear regression model that contains multiple categorical explanatory variables (e.g., cities, marital status), including other binary and continous variables. The output is 0/1 ...
1
vote
0answers
15 views

Encode categorical variables with many labels

I am trying to predict a multiclass categorical outcome variable by comparing different classifier algorithms. I've got a dataset that includes two categorical variables that have many labels (>...
0
votes
0answers
6 views

How to avoid categorisation of complex response?

Imagine I want to model which factors affect Melanoma cancer staging on diagnosis (short description: there are many categories and sub-categories and several measurements such as size/depth/lymph ...
0
votes
0answers
11 views

How to compute the new odds ratios and standard errors when changing the reference level of a categorical variable

I have a binary logistic regression that has a multi-level categorical factor (e.g., colors). Each level in the factor has a B and standard error (SE), from which I can compute the odds ratio (OR) ...
0
votes
0answers
14 views

Recoding income variables in trends analysis

I will greatly appreciate your advice. I will like to control for family income in my analysis of trends in self-rated health between year 2000 and 2018. The NHIS data from IPUMS that I'm using has ...
0
votes
0answers
12 views

Number of levels of categorical variable in Multilevel-Model

There is a rule of thumb that each prediction parameter in a regression must be supported by 10-15 observations. If I use dummy coding to represent the categorical variable in a multilevel-model, do ...
0
votes
0answers
33 views

Options when model complexity and separation causes non-convergence in logistic regression

I have created an example data set here My data represent the presence/absence of a particular animal species (data$outcome) and measurements of trees. I would like ...
0
votes
1answer
48 views

How to handle the dummy variables with overlapping categories?

Background of The Question Let's say, I have four categories (A, B, C, D). Considering one (D) as a reference variable, there will be three categories on which I have to work. But the problem is one ...
5
votes
4answers
108 views

How to visualise data where one variable is continuous and the other is categorical?

This question is very simple but I have been struggling in getting the right script for this. My data set goes as follows: ...
0
votes
2answers
57 views

Performing regression on a dataset with lots of categories

I am trying to work on a price prediction model, the attributes have lots of categories and all these categories are coded as integers. I am assuming if I build a regression model on this, the model ...
0
votes
1answer
12 views

spatially explicit longitudinal categorical change response to predictor variables

I'm doing a land use/land cover change (LULCC) analysis with annual data spanning 10 years. The land cover class pixels change annually (ie A -> B -> D -> A). There are 5 nominal response ...
1
vote
2answers
146 views

Target Encoder for Logistic regression

I have a data set that has many categorical values, I want to build a linear model using Logistic regression algorithm. One way of handling Categorical variables is ...
1
vote
2answers
617 views

Optimal binning methods for categorical variables

I'm running a multinomial logit to predict the outcome of a categoric response variable. I have both continuous and categoric independent variables, and I know it's bad practicde to bin the ...
0
votes
1answer
61 views

Higher Order Category Overlap Analysis

I am attempting to analyse the categorical overlap of a dataset to ultimately ascertain the optimal way of categorising the data to minimise the amount of used categories to describe the dataset. ...
3
votes
1answer
73 views

Does it have any meaning to compute the $\chi^2$ and the exact Fisher test on big contingency tables

I have several datasets containing integers. I want to perform a bivariate analysis between a specific subset of variables. However, some of them have a lot of modalities. Is computing a $\chi^2$ ...
0
votes
0answers
44 views

Sampling Technique: Categorical Data, Many Levels

I have a data set that has a categorical variable with almost half the number of observations as categories. Certain categories have only one observation. A minimal reproducable example in R would ...
1
vote
0answers
28 views

Converting Continuous variable to Categorical [duplicate]

When should one consider converting continuous variable into categorical variable ? Are there guidelines ? Is it justified to bin skewed variable ? How should I determine the range / binning when I do ...
2
votes
1answer
28 views

Does this violate the assumption of independence for regression

A very basic question which I have never encountered a discussion of before. I am conducting bivariate logistic regression (although my question applies to linear models as well). I have 11,500 ...
-1
votes
1answer
59 views

too much levels in the categorical variable in a GLM

I have 187 observations, the categorical variable is a predictor. My response variable is CPUE (catch per unit of effort). My goal is to know which of these variables (temperature, chlorophyll, depth, ...
1
vote
2answers
373 views

Performing one-hot encoding on a very large dataset

I am currently analysis a data set containing 654281 observations and 27 variables. I aim to perform binary logistic regression and many of my variables are categorical. I know one hot encoding is ...
1
vote
1answer
50 views

How to productionize a k-fold target-encoded feature?

I am attempting to build a model that has many predictors which are both categorical and large in cardinality. Target encoding looks to be a good solution for including these features, but I'm unsure ...
0
votes
0answers
26 views

How to deal with 100+ levels in categorical variable in multiple linear regression? [duplicate]

Im trying to model: Y~x0+x1+x2+x3+x4, were Y is a continous variable (cost), x0 is the ...
1
vote
1answer
42 views

Select top-k feature from a categorical variable using $\chi^2$

I am working with a categorical variable that has a lot of levels (let's say more than 20). I would like to binarize all the levels doing one-hot-encoding in order to use these new variables in a ...
0
votes
1answer
33 views

Statistical significance of a relationship of two categorical fields with more than two classes

From my dataset, I have two columns called the cuisine and the restaurant-grade. Each column corresponds to a restaurant. There are 6 different cuisines and 5 different grades. The question that I am ...
0
votes
1answer
122 views

Dealing with over 1000 categorical values (which are also a unique identifiers)

I am preparing my dataset for a logistic regression and need to check how best to handle a column with categorical values. As the dataset is for sales transactions, the column in question is the ...
1
vote
0answers
152 views

Handling rare levels in a categorical variable? (or maybe it's not categorical at all)

I have a dataset where I'm trying to predict completion time of an application. There are a number of numeric and categorical predictors, with a one group of predictors being holds. An application may ...
1
vote
1answer
56 views

Statistical measures for variable selection?

I have a data set which has multiple features (26) of high cardinality (categorical), low cardinality (categorical), numerical type. I wanted to select features for the target (numerical) prediction. ...
1
vote
0answers
29 views

Amount of information a row adds to a data set

I'm sorry if this is vaguely worded, but I'm looking for a way to score each row in a dataset by, essentially, the amount of information that that row adds - or the uniqueness of that row in the data ...
0
votes
0answers
33 views

Estimating coefficients of a large categorical variable

I'm trying to fit a GLM model with a categorical variable with 400 categories, and I'd like to reduce the number of categories. There are some categories with a lot of data, and a lot of categories ...
1
vote
1answer
328 views

What happens when you merge dummy variables together?

Suppose I want to regress $X$ on $y$, controlling for categorical $z$ with $100$ different levels. I believe that linear regression is appropriate. Normally I would create dummies $D_i$ for each ...
4
votes
1answer
615 views

Alternatives to using dummy variables?

I am working on this dataset: https://www.kaggle.com/russellyates88/suicide-rates-overview-1985-to-2016, and it has a lot of categorical variables, while I am more used to work with the continuous ...
0
votes
0answers
139 views

Using label encoder on a categorical feature that we want to embed

I have a dataset with feature that have very high cardinality, doing one-hot encoding is not an option because of memory limitations, so I am currently label encoding this feature and then I feed that ...
2
votes
0answers
177 views

Gradient boosting (GB) splitting methods (categorical features)

Regarding categorical features - ordinary trees treat categorical features in two main ways, CART - considers only binary splitting, those computing the mean response value (y_mean_i per each category ...
1
vote
2answers
554 views

How to correctly account for country effects in logistic regression?

I use a database with entries at firm-level in 12 countries in 2008. I try to estimate innovation (0/1) based on few firm-level variables. I also want to see if / how much innovation is also due to ...
0
votes
1answer
37 views

How to deal with categorical independent variables with numerous levels [duplicate]

How to deal with regression when most of the independent variables are categorical having numerous (more than 10) levels and the dependent variable is continuous? For this would it make sense to ...
0
votes
0answers
51 views

Categorical values too many Levels [duplicate]

I am working on something there trying to predict a cost per location there are 8 variables one of them is a categorical value that has over 300 levels of postal codes in the entire provinces will ...
1
vote
1answer
33 views

Logit model for hundres of items - can and should I use the items as a category variable?

I am in the early phase of a new project about looking at multiple factors that potentially influence the probability that an item fails quality inspection. I am interested in seeing whether each ...
1
vote
1answer
44 views

Is there a point where you wouldn't use dummy variables? I.e., if getting dummy vars would lead to hundreds of variables? [duplicate]

I built a web scraper that drew in a bunch of data and I have more qualitative variables than I expected. Originally there were just a few quantitative variables that I had intended to consider but, ...
2
votes
0answers
33 views

Clustering of very high dimensional data and large number of examples without losing info in dimensions

I'm trying to get a grasp on scalability of clustering algorithms, and have a toy example in mind. Let's say I have around a million or so songs from $50$ genres. Each song has characteristics - some ...
0
votes
0answers
39 views

Help in Handling multiple classes in independent categorical variables and improve performance

The dataset has 4 categorical and 1 numerical variable and a timestamp variable. Out of 4, three categorical variables are having more than 100 categories. I tried doing one-hot encoding on the whole ...
4
votes
1answer
1k views

Encoding IP Address as a Predictor in Machine Learning

Is there some approach to "encoding" IP Address (IPv4) in a way that the new representation can capture both cardinality and the statistical distribution of the full range of IP address and also ...
2
votes
1answer
630 views

Is it a good practice to drop rare categorical data?

I have a dataset with about 100K samples described mostly by categorical features. The number of unique values in the categories range from 20 to almost 7000. Since these are categorical values and ...
9
votes
1answer
12k views

Encoding of categorical variables with high cardinality

For unsupervised anomaly detection / fraud analytics on credit card data (where I don't have labeled fraudulent cases), there are a lot of variables to consider. The data is of mixed type with ...
0
votes
0answers
97 views

Should we penalize dummy variables? [duplicate]

Using glmnet we run the following regression cvfit = cv.glmnet(x,y, alpha = 0, intercept = FALSE) where $y$ is the response variable and $x$ is the input matrix....
0
votes
0answers
40 views

Random Forest limitation of 53 categories

Since Random Forest has limitation of 53 categories, which categorical model can be applied to categorical data with vectors which have 100K+ levels? ...
0
votes
0answers
37 views

Condensing values of categorical data

Beginner ML question here. I have a dataframe with a categorical column, a lot of the values are slightly different but essentially mean the same thing. Here's an example of such values: ...
1
vote
2answers
66 views

Finding Relationship between Categorical and Continuous data

A subset of my dataset looks as follows where cells in "cat1_ids" column contains list of "cat1" categories and cells in "person_id_list" column contains list of persons id. There are 2000 "cat1" ...
2
votes
1answer
523 views

Classification with ONLY categorical data

Suppose I have a table with some factor characteristics of some plants. For instance, petal color, pollen color, and so on. What is the best way to classify that data? Is it feasible to use some of ...

1
2 3 4 5