A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Questions tagged [many-categories]

Categorical variables with large number of levels, and statistical methods for working with such variables (example: fused lasso).

Filter by
Sorted by
Tagged with
1
vote
1answer
37 views

Alternatives to using dummy variables?

I am working on this dataset: https://www.kaggle.com/russellyates88/suicide-rates-overview-1985-to-2016, and it has a lot of categorical variables, while I am more used to work with the continuous ...
0
votes
0answers
26 views

Modelling distance distributions

Overview I am building a model for a dependent variable which represents distance (from a more general perspective, my response can only take positive values). Moreover, it is imperative that I am ...
0
votes
0answers
15 views

Gradient boosting (GB) splitting methods (categorical features)

Regarding categorical features - ordinary trees treat categorical features in two main ways, CART - considers only binary splitting, those computing the mean response value (y_mean_i per each category ...
0
votes
0answers
14 views

Random Forest with high number of categorical features?

I have a housing market data frame with the columns house_price, surface_area, bedrooms, rental_agency, postcode ,furnished, inclusive and I have a university ...
1
vote
2answers
87 views

How to correctly account for country effects in logistic regression?

I use a database with entries at firm-level in 12 countries in 2008. I try to estimate innovation (0/1) based on few firm-level variables. I also want to see if / how much innovation is also due to ...
0
votes
1answer
30 views

How to deal with categorical independent variables with numerous levels [duplicate]

How to deal with regression when most of the independent variables are categorical having numerous (more than 10) levels and the dependent variable is continuous? For this would it make sense to ...
0
votes
0answers
47 views

Categorical values too many Levels

I am working on something there trying to predict a cost per location there are 8 variables one of them is a categorical value that has over 300 levels of postal codes in the entire provinces will ...
1
vote
1answer
28 views

Logit model for hundres of items - can and should I use the items as a category variable?

I am in the early phase of a new project about looking at multiple factors that potentially influence the probability that an item fails quality inspection. I am interested in seeing whether each ...
1
vote
1answer
32 views

Is there a point where you wouldn't use dummy variables? I.e., if getting dummy vars would lead to hundreds of variables? [duplicate]

I built a web scraper that drew in a bunch of data and I have more qualitative variables than I expected. Originally there were just a few quantitative variables that I had intended to consider but, ...
0
votes
0answers
25 views

How do I combine the categories of a classification system to find the maximum reliability?

I described my study design in another question. Briefly, I recruited 16 unique raters to classify 35 items according to a seven-category classification system. I measured the interrater reliability ...
0
votes
0answers
19 views

Help in Handling multiple classes in independent categorical variables and improve performance

The dataset has 4 categorical and 1 numerical variable and a timestamp variable. Out of 4, three categorical variables are having more than 100 categories. I tried doing one-hot encoding on the whole ...
4
votes
1answer
254 views

Encoding IP Address as a Predictor in Machine Learning

Is there some approach to "encoding" IP Address (IPv4) in a way that the new representation can capture both cardinality and the statistical distribution of the full range of IP address and also ...
2
votes
1answer
150 views

Is it a good practice to drop rare categorical data?

I have a dataset with about 100K samples described mostly by categorical features. The number of unique values in the categories range from 20 to almost 7000. Since these are categorical values and ...
2
votes
1answer
3k views

Encoding of categorical variables with high cardinality

For unsupervised anomaly detection / fraud analytics on credit card data (where I don't have labeled fraudulent cases), there are a lot of variables to consider. The data is of mixed type with ...
0
votes
0answers
63 views

Should we penalize dummy variables? [duplicate]

Using glmnet we run the following regression cvfit = cv.glmnet(x,y, alpha = 0, intercept = FALSE) where $y$ is the response variable and $x$ is the input matrix....
0
votes
0answers
36 views

Random Forest limitation of 53 categories

Since Random Forest has limitation of 53 categories, which categorical model can be applied to categorical data with vectors which have 100K+ levels? ...
0
votes
0answers
34 views

Condensing values of categorical data

Beginner ML question here. I have a dataframe with a categorical column, a lot of the values are slightly different but essentially mean the same thing. Here's an example of such values: ...
1
vote
2answers
60 views

Finding Relationship between Categorical and Continuous data

A subset of my dataset looks as follows where cells in "cat1_ids" column contains list of "cat1" categories and cells in "person_id_list" column contains list of persons id. There are 2000 "cat1" ...
2
votes
1answer
65 views

Classification with ONLY categorical data

Suppose I have a table with some factor characteristics of some plants. For instance, petal color, pollen color, and so on. What is the best way to classify that data? Is it feasible to use some of ...
3
votes
1answer
71 views

Forecasting sales (in units) for thousand of products

I've got into this internship in a retail company and they asked me to think a way to forecast their daily sales (in units) in all their stores (with thousands of skus each one). At first I thought ...
0
votes
1answer
17 views

Testing an intervention by repeated measure of opinion before and after the intervention

How can we test the null hypothesis that an intervention has no effect on opinions for 44 people. These opinions were measured as frequencies in 3 categories (A, B, C) before and after the ...
0
votes
0answers
32 views

How to deal with a potentially multiple categorical variable

I'm building a model that has, as inputs, some categorical variables. I had already dealt with this sort of data before, and applied different techniques as creation of dummy variables and factor ...
1
vote
1answer
130 views

Linear Regression and High Dimensional Categorical Data

I've read that mean encoding is useful for classification tasks with high dimensional categorical data. My question: What kinds of encodings are effective for high dimensional categorical data in ...
1
vote
1answer
76 views

How to “choose” categorical variables which have impact in a regression?

I have a dataset of about 50K samples. I have approximately 90 columns which are all categorical and they're used to predict a price. There's no other continuous value. I'm trying to select "which of ...
0
votes
1answer
85 views

How can I make use of zip codes when I am building a model for fraud detection

I have gone through few articles but I am not convinced on what should I do with these. I know from business standpoint it might be good to consider fraudulent transactions happening from unknown ...
2
votes
1answer
77 views

Multiple categorical IVs and DVs with 3/5 levels prediction model

I have a data-set with 8 categorical IVs with 2/3 levels (one level for one type of conditions), 2 categorical DVs with 3/5 levels (one level for one type of responses:dis/even/ad). Participants ...
0
votes
0answers
23 views

Visual representation of strong associations of two categorical variables

I have a dataset of one categoric variable "supermarket" for each individual person and multiple "product" categories per person and supermarket. E.g.: Person 1 went shopping in supermarket X and ...
1
vote
0answers
115 views

Xgboost / Boosted decision trees: Representing categorical id numbers as continuous integer variable

I've been reading through some kernels at kaggle.com for a sales forecasting competition, and noticed that a lot of people using Xgboost are feeding it categorial ID variables, represented as ...
1
vote
0answers
60 views

(multiple) fractional outcomes & autoregression

Let me start with a broad description of the problem and I will then describe my approach (that might be totally inappropiate). The big goal is to predict the distribution of population of a given age ...
1
vote
0answers
25 views

How to compare frequencies of categorical variable with 3 possible values

There is one variable which can get one of 3 values and one sample. Lets assume values are A, B, C and frequencies are x, y, and z. How could I find if x > max(y,z), statistically significant? Or, in ...
3
votes
2answers
423 views

Random Forest Regression with sparse data in Python

I am working on a Random Forest regression model to predict housing prices. I have about 500k rows of data with the following information: 1.House area in square meters. 2.Number of rooms. 3.City. ...
0
votes
0answers
50 views

Compare frequencies in HUGE number of categories

I have 2 samples, each with many (e.g. 100'000) different categories (colors in urn model), and counts for each category (number of balls of each color - many with few counts, and few with higher ...
0
votes
1answer
391 views

One-hot-encoding gives untractable amount of classes

I'm performing regression on the price of bycicles based on their brand, model and submodel. These features are hierarchical: one model belongs only to one brand but one brand can have many models. ...
2
votes
4answers
982 views

Regression with Lots of Categorical Variables

I'm facing a regression task with many categorical and few numeric features. I encoded them into dummies and removed the first dummy column for each feature. I am not getting very good R2 at all. I ...
1
vote
1answer
209 views

How to handle large number of categories in a regression model? [duplicate]

I have a dataset with 200,000 entries with four columns (time_of_day, order_size, time_taken, shop_number). I need to build a model and predict time_taken using the other three variables. There are ...
0
votes
0answers
38 views

Should I build a different model for each subset

I have a dataset which has categorical variable class and it has around 10 classes in it. I am trying to solve a regression problem I am not understanding whether I should build a model on entire ...
3
votes
1answer
368 views

Why it might be bad to have too many feature levels

I am aware that a feature with too many levels might be bad for a number of algorithms (e.g. Logistic Regression). A typical approach to fix this would be to group the categories with a frequency ...
1
vote
1answer
92 views

Encoding variable number of categorical features

I have a dataset listing the software installed for each user. This dataset shall be used (in conjuction with other user datasets) to classify the user into 4 (imbalanced) categories. There are over ...
1
vote
1answer
59 views

ICD10 (categorical) encoding

I am trying to figure out how best to encode ICD10 codes for input into a machine learning model. It isn't ordinal by any means, however, there is a sort of logic you can apply to just the labels ...
0
votes
1answer
29 views

text mining - vocabulary size very large

Question: when you have create a corpus of let’s say, 10,000 documents, and the vocabulary size made for these is let’s say, 1 million, what best practices exist to either work with this type of ...
2
votes
1answer
34 views

Categorical variable---Census tracts in Manhattan

I have a categorical variable, geoid's for census tracts in Manhattan, with 288 levels. After running a linear regression on my categorical variable and other predictors (population, weather, ...) I ...
5
votes
1answer
132 views

How to build a predictive model when more levels of a categorical predictor are possible than appear in the training data

I have created an Artificial Neural Network with 4 categorical features and a binary outcome either 1 for suspicious or 0 for non-suspicious: ...
3
votes
1answer
3k views

How to handle too many categorical features with too many categories for XGBoost?

In my data I have 35 features and 14 of them are categorical. Half of them have 3 to 4 categories but others have 14 to 28 categories. One Hot Encoding them would only lead to a sparse matrix with ...
2
votes
1answer
649 views

Encoding of categorical data/feature/predictor for binary classification

ML newbie here, currently looking at a binary classification problem. I have quite a good number of training data (easily over 50k) which consists of both numeric and categorical data. The categorical ...
0
votes
1answer
99 views

how to deal with categorical features (with distinct 10000+ values) other than conversion to one-hot encode and ordinal

Machine Learning Problem : I have a doubt in one of my feature which has an categorical value 1. One way of dealing with it would be like converting those values into numbers means in ordinal form. ...
1
vote
0answers
396 views

Dealing with categorical feature for xgboost using sagemaker

Currently, I have a dataset which contains 200,000+ datapoints and it contains 20 features with ~10 features as categorical. These categorical columns are countries, state, localities which contains >...
0
votes
1answer
60 views

Modeling number of spectators in football

Can anyone give ideas on the possible best way forward to solve this specific machine learning problem for sports analytics? Data set looks like: ...
3
votes
1answer
1k views

Too many dummy variable in regression model

we have about 50000 models of mobile phone (like Galaxy S7, iPhone 9) in database and the size of data is about 3 million. We want to find the mobile phones that have the least call success rate ( ...
2
votes
1answer
122 views

How to improve accuracy in the case of categorical data with many levels and no correlation

Consider a simple multiclass problem in which there is a categorical variable with many levels (>1000). The nature of the problem is such that we can not reduce the dimensions of this variable. The ...
1
vote
2answers
57 views

Big categorical data

I am trying to predict the price of used vehicles using three different models: Regression, ANN, and random forest. I am having 6 variables as an input for my model. One of my variables is the ...