Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [many-categories]

Categorical variables with large number of levels, and statistical methods for working with such variables (example: fused lasso).

1
vote
2answers
45 views

Finding Relationship between Categorical and Continuous data

A subset of my dataset looks as follows where cells in "cat1_ids" column contains list of "cat1" categories and cells in "person_id_list" column contains list of persons id. There are 2000 "cat1" ...
0
votes
0answers
21 views

Classification with ONLY categorical data

Suppose I have a table with some factor characteristics of some plants. For instance, petal color, pollen color, and so on. What is the best way to classify that data? Is it feasible to use some of ...
0
votes
0answers
40 views

Many-hot encoding for Hamming codes using 1D convolutional autoencoder

I am trying a simple experiment where I take 16 numbers [0 to 15] that represent the Hamming codebook, and try to reconstruct it using an autoencoder. Instead of using a one-hot encoding scheme (16 ...
2
votes
1answer
25 views

Forecasting sales (in units) for thousand of products

I've got into this internship in a retail company and they asked me to think a way to forecast their daily sales (in units) in all their stores (with thousands of skus each one). At first I thought ...
0
votes
0answers
8 views

Testing an intervention by repeated measure of opinion before and after the intervention

How can we test the null hypothesis that an intervention has no effect on opinions for 44 people. These opinions were measured as frequencies in 3 categories (A, B, C) before and after the ...
0
votes
0answers
22 views

How to deal with a potentially multiple categorical variable

I'm building a model that has, as inputs, some categorical variables. I had already dealt with this sort of data before, and applied different techniques as creation of dummy variables and factor ...
1
vote
1answer
35 views

How to “choose” categorical variables which have impact in a regression?

I have a dataset of about 50K samples. I have approximately 90 columns which are all categorical and they're used to predict a price. There's no other continuous value. I'm trying to select "which of ...
0
votes
0answers
22 views

How can I make use of zip codes when I am building a model for fraud detection

I have gone through few articles but I am not convinced on what should I do with these. I know from business standpoint it might be good to consider fraudulent transactions happening from unknown ...
2
votes
1answer
47 views

Multiple categorical IVs and DVs with 3/5 levels prediction model

I have a data-set with 8 categorical IVs with 2/3 levels (one level for one type of conditions), 2 categorical DVs with 3/5 levels (one level for one type of responses:dis/even/ad). Participants ...
0
votes
0answers
19 views

Visual representation of strong associations of two categorical variables

I have a dataset of one categoric variable "supermarket" for each individual person and multiple "product" categories per person and supermarket. E.g.: Person 1 went shopping in supermarket X and ...
1
vote
0answers
59 views

Xgboost / Boosted decision trees: Representing categorical id numbers as continuous integer variable

I've been reading through some kernels at kaggle.com for a sales forecasting competition, and noticed that a lot of people using Xgboost are feeding it categorial ID variables, represented as ...
1
vote
0answers
58 views

(multiple) fractional outcomes & autoregression

Let me start with a broad description of the problem and I will then describe my approach (that might be totally inappropiate). The big goal is to predict the distribution of population of a given age ...
1
vote
0answers
18 views

How to compare frequencies of categorical variable with 3 possible values

There is one variable which can get one of 3 values and one sample. Lets assume values are A, B, C and frequencies are x, y, and z. How could I find if x > max(y,z), statistically significant? Or, in ...
0
votes
0answers
40 views

Compare frequencies in HUGE number of categories

I have 2 samples, each with many (e.g. 100'000) different categories (colors in urn model), and counts for each category (number of balls of each color - many with few counts, and few with higher ...
0
votes
1answer
73 views

One-hot-encoding gives untractable amount of classes

I'm performing regression on the price of bycicles based on their brand, model and submodel. These features are hierarchical: one model belongs only to one brand but one brand can have many models. ...
1
vote
1answer
80 views

How to handle large number of categories in a regression model? [duplicate]

I have a dataset with 200,000 entries with four columns (time_of_day, order_size, time_taken, shop_number). I need to build a model and predict time_taken using the other three variables. There are ...
0
votes
0answers
29 views

Should I build a different model for each subset

I have a dataset which has categorical variable class and it has around 10 classes in it. I am trying to solve a regression problem I am not understanding whether I should build a model on entire ...
0
votes
1answer
139 views

Why it might be bad to have too many feature levels

I am aware that a feature with too many levels might be bad for a number of algorithms (e.g. Logistic Regression). A typical approach to fix this would be to group the categories with a frequency ...
0
votes
0answers
34 views

Encoding variable number of categorical features

I have a dataset listing the software installed for each user. This dataset shall be used (in conjuction with other user datasets) to classify the user into 4 (imbalanced) categories. There are over ...
1
vote
1answer
44 views

ICD10 (categorical) encoding

I am trying to figure out how best to encode ICD10 codes for input into a machine learning model. It isn't ordinal by any means, however, there is a sort of logic you can apply to just the labels ...
0
votes
1answer
13 views

text mining - vocabulary size very large

Question: when you have create a corpus of let’s say, 10,000 documents, and the vocabulary size made for these is let’s say, 1 million, what best practices exist to either work with this type of ...
3
votes
1answer
1k views

How to handle too many categorical features with too many categories for XGBoost?

In my data I have 35 features and 14 of them are categorical. Half of them have 3 to 4 categories but others have 14 to 28 categories. One Hot Encoding them would only lead to a sparse matrix with ...
2
votes
1answer
294 views

Encoding of categorical data/feature/predictor for binary classification

ML newbie here, currently looking at a binary classification problem. I have quite a good number of training data (easily over 50k) which consists of both numeric and categorical data. The categorical ...
0
votes
1answer
70 views

how to deal with categorical features (with distinct 10000+ values) other than conversion to one-hot encode and ordinal

Machine Learning Problem : I have a doubt in one of my feature which has an categorical value 1. One way of dealing with it would be like converting those values into numbers means in ordinal form. ...
1
vote
0answers
246 views

Dealing with categorical feature for xgboost using sagemaker

Currently, I have a dataset which contains 200,000+ datapoints and it contains 20 features with ~10 features as categorical. These categorical columns are countries, state, localities which contains >...
0
votes
1answer
57 views

Modeling number of spectators in football

Can anyone give ideas on the possible best way forward to solve this specific machine learning problem for sports analytics? Data set looks like: ...
3
votes
1answer
541 views

Too many dummy variable in regression model

we have about 50000 models of mobile phone (like Galaxy S7, iPhone 9) in database and the size of data is about 3 million. We want to find the mobile phones that have the least call success rate ( ...
2
votes
1answer
41 views

How to improve accuracy in the case of categorical data with many levels and no correlation

Consider a simple multiclass problem in which there is a categorical variable with many levels (>1000). The nature of the problem is such that we can not reduce the dimensions of this variable. The ...
0
votes
1answer
41 views

Big categorical data

I am trying to predict the price of used vehicles using three different models: Regression, ANN, and random forest. I am having 6 variables as an input for my model. One of my variables is the ...
0
votes
0answers
33 views

Clustering industry field

I have a big company dataset with an industry variable that looks like this: ...
0
votes
1answer
30 views

How to preprocess the Categorical Data with large number of columns

I have large number of categorical columns in my dataset, I want to preprocess the data, I know that I have to do one hot encoding but in data set columns or not in specific order they are at random ...
0
votes
1answer
14 views

Account for small but significant categories in model

I want to model participation to a campaign. I have ~200 variables for ~100k observations. Many variables are categorical and I often found high participation rates in smaller categories, for ...
1
vote
1answer
1k views

Encoding categorical variables with hundreds of levels for machine learning algorithms?

Despite the fact that this problem is probably overdone and heavily researched, I want to use machine learning to solve problems related to basketball (predicting which team will win the game, will a ...
1
vote
0answers
24 views

Variable selection with tree-structured covariates?

Let's say I want to do regression and that there's a categorical variable which has an inherent tree structure. Using an example from my field of linguistics, let's say I'm trying to predict a binary ...
2
votes
1answer
80 views

What kinds of algorithms work well with hundreds of thousands of output classes?

I have a limited amount of data for each class (about 100 samples), but I have about 100000 such classes. What kind of classification algorithm would work on this? (Apart from a NN with hierarchical ...
1
vote
0answers
1k views

Classification with 500 Categories

Currently I am working on several projects with classification algorithms. The number of categories is very high (between 100 and 4 000, but let us assume it is 500). Which algorithms are suitable ...
1
vote
1answer
52 views

Getting rid of categorical features

I am working on training a reinforcement learning agent with a huge dataset of past human players' experience. Each user had to independently increase their game score. I am using the dataset because ...
2
votes
0answers
391 views

Fitting multilevel categorical variables with neural nets

Most of the neural net algorithms I'm aware of require multilevel, ANOVA-type categorical features to be preprocessed into a set of dummy (0,1) variables. So, if one has a single categorical feature ...
3
votes
1answer
120 views

Effective data visualizations for large numbers of classes

Experimenting with a few approaches to visualize error modes/rates for models with a large number (100-500) of classes. For smaller numbers of classes have been using ROC graphs (with each class ...
1
vote
0answers
426 views

Encoding categorical variables with lots of categories [duplicate]

I recently had to build a model for flight delay predictions. The data has multiple columns of categorical variables and some continuous variables. For example: Airline, Arriving airport, Temperature,...
0
votes
1answer
70 views

Question on Multi-level Categorical Variable

I have a question on the coding/use of categorical variables: In short, I have two control variables in an SPSS regression model. One is participant (15 levels, or participants) and one other ...
0
votes
0answers
46 views

Classification of 20 million records to 200 categories

I have a data set of 20 million records, which I need to assign to 200 categories. (Business language - based on the text description, I need to assign it to one of the 200 departments). Suppose I ...
3
votes
1answer
52 views

Maximum Number of Categorical Outcomes for Multinomial Logistic Regression?

Based on 4 or 5 predictor variables, I would like to determine "affinity" to a group of 26 different non-profits that an individual could potentially donate money to. I have approximately 1 million ...
0
votes
0answers
172 views

On what basis can we combine levels in a factor variable when the target variable is binary? [duplicate]

I am working on a dataset in which a variable has following levels Levels: 0 1 2 3 4 5 8 Frequency: 608 209 28 16 18 5 7 The target variable is binary....
4
votes
0answers
425 views

One hot encoding alternatives for large categorical values? [duplicate]

Hi have dataframe with large categorical values over 1600 categories is there any way I can find alternatives so that I don't have over 1600 columns. I found this below interesting link http://...
1
vote
1answer
284 views

Dummy coding categorical variables with lots of unique values using log2?

I'm trying to understand the logic behind this binary encoder. It automatically takes categorical variables and dummy codes them (similar to one-hot-encoding on sklearn), but reduces the number of ...
1
vote
0answers
220 views

Modeling with categorical variables where observations fall into multiple categories?

I'm running multivariate mixed effects linear meta-analytic models in R. One of my predictor variables is categorical, but each observation generally has multiple categorical values. I'm not sure how ...
0
votes
0answers
2k views

Logistic Regression with Categorical Variables: 1: glm.fit: algorithm did not converge 2: glm.fit: fitted probabilities numerically 0 or 1 occurred

In my model, I have a Response variable, 0s or 1s. I have 15 categorical variables, some of which have 150+ levels. Should I potentially exclude them from my model? When I run ...
1
vote
0answers
29 views

dropping number of levels in nominal features [duplicate]

I do not have a good theoretical background in machine learning, therefore will ask a maybe naive question, nonetheless, it's important for implementing my learning algorithm. I'm having a question ...
1
vote
0answers
380 views

Classification into a High number of Classes

Consider a classification task wherein the training data has around 100,000 sentences with around 1000 labels. These 100,000 sentences can possibly be grouped hierarchically. The task at hand is given ...