Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [many-categories]

Categorical variables with large number of levels, and statistical methods for working with such variables (example: fused lasso).

1
vote
1answer
22 views

How to “choose” categorical variables which have impact in a regression?

I have a dataset of about 50K samples. I have approximately 90 columns which are all categorical and they're used to predict a price. There's no other continuous value. I'm trying to select "which of ...
0
votes
0answers
16 views

How can I make use of zip codes when I am building a model for fraud detection

I have gone through few articles but I am not convinced on what should I do with these. I know from business standpoint it might be good to consider fraudulent transactions happening from unknown ...
0
votes
0answers
6 views

How to Code & analyze multiple response questions in SPSS? [closed]

After exporting my data from Qualtrics to SPSS. The multiple answer options did not fit as assumed into the variable view. Instead of having (age) IV --> channel (DV) with several options, it ...
2
votes
1answer
39 views

Multiple categorical IVs and DVs with 3/5 levels prediction model

I have a data-set with 8 categorical IVs with 2/3 levels (one level for one type of conditions), 2 categorical DVs with 3/5 levels (one level for one type of responses:dis/even/ad). Participants ...
0
votes
0answers
18 views

Visual representation of strong associations of two categorical variables

I have a dataset of one categoric variable "supermarket" for each individual person and multiple "product" categories per person and supermarket. E.g.: Person 1 went shopping in supermarket X and ...
1
vote
0answers
41 views

Xgboost / Boosted decision trees: Representing categorical id numbers as continuous integer variable

I've been reading through some kernels at kaggle.com for a sales forecasting competition, and noticed that a lot of people using Xgboost are feeding it categorial ID variables, represented as ...
1
vote
0answers
58 views

(multiple) fractional outcomes & autoregression

Let me start with a broad description of the problem and I will then describe my approach (that might be totally inappropiate). The big goal is to predict the distribution of population of a given age ...
1
vote
0answers
17 views

How to compare frequencies of categorical variable with 3 possible values

There is one variable which can get one of 3 values and one sample. Lets assume values are A, B, C and frequencies are x, y, and z. How could I find if x > max(y,z), statistically significant? Or, in ...
0
votes
0answers
39 views

Compare frequencies in HUGE number of categories

I have 2 samples, each with many (e.g. 100'000) different categories (colors in urn model), and counts for each category (number of balls of each color - many with few counts, and few with higher ...
0
votes
1answer
35 views

One-hot-encoding gives untractable amount of classes

I'm performing regression on the price of bycicles based on their brand, model and submodel. These features are hierarchical: one model belongs only to one brand but one brand can have many models. ...
1
vote
1answer
64 views

How to handle large number of categories in a regression model? [duplicate]

I have a dataset with 200,000 entries with four columns (time_of_day, order_size, time_taken, shop_number). I need to build a model and predict time_taken using the other three variables. There are ...
0
votes
0answers
26 views

Should I build a different model for each subset

I have a dataset which has categorical variable class and it has around 10 classes in it. I am trying to solve a regression problem I am not understanding whether I should build a model on entire ...
0
votes
1answer
75 views

Why it might be bad to have too many feature levels

I am aware that a feature with too many levels might be bad for a number of algorithms (e.g. Logistic Regression). A typical approach to fix this would be to group the categories with a frequency ...
0
votes
0answers
30 views

Encoding variable number of categorical features

I have a dataset listing the software installed for each user. This dataset shall be used (in conjuction with other user datasets) to classify the user into 4 (imbalanced) categories. There are over ...
0
votes
1answer
10 views

text mining - vocabulary size very large

Question: when you have create a corpus of let’s say, 10,000 documents, and the vocabulary size made for these is let’s say, 1 million, what best practices exist to either work with this type of ...
3
votes
1answer
686 views

How to handle too many categorical features with too many categories for XGBoost?

In my data I have 35 features and 14 of them are categorical. Half of them have 3 to 4 categories but others have 14 to 28 categories. One Hot Encoding them would only lead to a sparse matrix with ...
2
votes
1answer
220 views

Encoding of categorical data/feature/predictor for binary classification

ML newbie here, currently looking at a binary classification problem. I have quite a good number of training data (easily over 50k) which consists of both numeric and categorical data. The categorical ...
0
votes
1answer
58 views

how to deal with categorical features (with distinct 10000+ values) other than conversion to one-hot encode and ordinal

Machine Learning Problem : I have a doubt in one of my feature which has an categorical value 1. One way of dealing with it would be like converting those values into numbers means in ordinal form. ...
1
vote
0answers
203 views

Dealing with categorical feature for xgboost using sagemaker

Currently, I have a dataset which contains 200,000+ datapoints and it contains 20 features with ~10 features as categorical. These categorical columns are countries, state, localities which contains >...
0
votes
1answer
56 views

Modeling number of spectators in football

Can anyone give ideas on the possible best way forward to solve this specific machine learning problem for sports analytics? Data set looks like: ...
3
votes
1answer
331 views

Too many dummy variable in regression model

we have about 50000 models of mobile phone (like Galaxy S7, iPhone 9) in database and the size of data is about 3 million. We want to find the mobile phones that have the least call success rate ( ...
2
votes
1answer
26 views

How to improve accuracy in the case of categorical data with many levels and no correlation

Consider a simple multiclass problem in which there is a categorical variable with many levels (>1000). The nature of the problem is such that we can not reduce the dimensions of this variable. The ...
0
votes
1answer
39 views

Big categorical data

I am trying to predict the price of used vehicles using three different models: Regression, ANN, and random forest. I am having 6 variables as an input for my model. One of my variables is the ...
0
votes
0answers
33 views

Clustering industry field

I have a big company dataset with an industry variable that looks like this: ...
0
votes
1answer
27 views

How to preprocess the Categorical Data with large number of columns

I have large number of categorical columns in my dataset, I want to preprocess the data, I know that I have to do one hot encoding but in data set columns or not in specific order they are at random ...
0
votes
1answer
14 views

Account for small but significant categories in model

I want to model participation to a campaign. I have ~200 variables for ~100k observations. Many variables are categorical and I often found high participation rates in smaller categories, for ...
1
vote
1answer
789 views

Encoding categorical variables with hundreds of levels for machine learning algorithms?

Despite the fact that this problem is probably overdone and heavily researched, I want to use machine learning to solve problems related to basketball (predicting which team will win the game, will a ...
1
vote
0answers
24 views

Variable selection with tree-structured covariates?

Let's say I want to do regression and that there's a categorical variable which has an inherent tree structure. Using an example from my field of linguistics, let's say I'm trying to predict a binary ...
2
votes
1answer
79 views

What kinds of algorithms work well with hundreds of thousands of output classes?

I have a limited amount of data for each class (about 100 samples), but I have about 100000 such classes. What kind of classification algorithm would work on this? (Apart from a NN with hierarchical ...
1
vote
0answers
1k views

Classification with 500 Categories

Currently I am working on several projects with classification algorithms. The number of categories is very high (between 100 and 4 000, but let us assume it is 500). Which algorithms are suitable ...
0
votes
0answers
18 views

Categorization with many categories: The most probable category or the most probable ratios of counts?

Suppose you are trying to classify observations into one of n categories using ordered multinomial logistic regression. Suppose further that one of the categories – category 1, say – is much more ...
1
vote
1answer
50 views

Getting rid of categorical features

I am working on training a reinforcement learning agent with a huge dataset of past human players' experience. Each user had to independently increase their game score. I am using the dataset because ...
0
votes
0answers
42 views

Can binary logistic regression be performed without a reference/baseline category?

I have different independent categorical variables with more than two categories, dependent variable is binomial. How do I compare the effect of these independent variables w.r.t an absolute reference ...
0
votes
0answers
13 views

Multiple categories, multiple data sets, over time

I am not a stats guy so forgive me if I don't use the terminology correctly. I have a group of 200 people. In Jan 2017, I measured them. In an Jan 2018 I measured them again. I want to know what ...
0
votes
0answers
91 views

Encoding categorical features of high cardinality [duplicate]

I'm building a logistic regression model and I have a number of categorical features some of which of very high cardinality. I was thinking of encoding them as dummies (n-1) but the number of features ...
2
votes
0answers
351 views

Fitting multilevel categorical variables with neural nets

Most of the neural net algorithms I'm aware of require multilevel, ANOVA-type categorical features to be preprocessed into a set of dummy (0,1) variables. So, if one has a single categorical feature ...
3
votes
1answer
108 views

Effective data visualizations for large numbers of classes

Experimenting with a few approaches to visualize error modes/rates for models with a large number (100-500) of classes. For smaller numbers of classes have been using ROC graphs (with each class ...
1
vote
0answers
424 views

Encoding categorical variables with lots of categories [duplicate]

I recently had to build a model for flight delay predictions. The data has multiple columns of categorical variables and some continuous variables. For example: Airline, Arriving airport, Temperature,...
0
votes
1answer
63 views

Question on Multi-level Categorical Variable

I have a question on the coding/use of categorical variables: In short, I have two control variables in an SPSS regression model. One is participant (15 levels, or participants) and one other ...
0
votes
0answers
42 views

Classification of 20 million records to 200 categories

I have a data set of 20 million records, which I need to assign to 200 categories. (Business language - based on the text description, I need to assign it to one of the 200 departments). Suppose I ...
3
votes
1answer
49 views

Maximum Number of Categorical Outcomes for Multinomial Logistic Regression?

Based on 4 or 5 predictor variables, I would like to determine "affinity" to a group of 26 different non-profits that an individual could potentially donate money to. I have approximately 1 million ...
0
votes
0answers
132 views

On what basis can we combine levels in a factor variable when the target variable is binary? [duplicate]

I am working on a dataset in which a variable has following levels Levels: 0 1 2 3 4 5 8 Frequency: 608 209 28 16 18 5 7 The target variable is binary....
4
votes
0answers
421 views

One hot encoding alternatives for large categorical values? [duplicate]

Hi have dataframe with large categorical values over 1600 categories is there any way I can find alternatives so that I don't have over 1600 columns. I found this below interesting link http://...
1
vote
1answer
245 views

Dummy coding categorical variables with lots of unique values using log2?

I'm trying to understand the logic behind this binary encoder. It automatically takes categorical variables and dummy codes them (similar to one-hot-encoding on sklearn), but reduces the number of ...
1
vote
0answers
210 views

Modeling with categorical variables where observations fall into multiple categories?

I'm running multivariate mixed effects linear meta-analytic models in R. One of my predictor variables is categorical, but each observation generally has multiple categorical values. I'm not sure how ...
0
votes
0answers
2k views

Logistic Regression with Categorical Variables: 1: glm.fit: algorithm did not converge 2: glm.fit: fitted probabilities numerically 0 or 1 occurred

In my model, I have a Response variable, 0s or 1s. I have 15 categorical variables, some of which have 150+ levels. Should I potentially exclude them from my model? When I run ...
1
vote
0answers
29 views

dropping number of levels in nominal features [duplicate]

I do not have a good theoretical background in machine learning, therefore will ask a maybe naive question, nonetheless, it's important for implementing my learning algorithm. I'm having a question ...
1
vote
0answers
311 views

Classification into a High number of Classes

Consider a classification task wherein the training data has around 100,000 sentences with around 1000 labels. These 100,000 sentences can possibly be grouped hierarchically. The task at hand is given ...
1
vote
1answer
345 views

Type of analysis for categorical variables with thousands of levels predicting one categorical variable?

I am attempting a classification problem where I am trying to assign a row of data to a category that has about 1,000 levels. My predictors are codes that are categorical and can be any of 10,000 ...
0
votes
1answer
870 views

how to handle large categorical values in data frame? [closed]

I have over 2 million records which I am using to build the machine learning model. But I have a column with categorical values over 1600. I cannot convert the data into 1600 1-dimensional columns <...