Questions tagged [categorical-encoding]

Representing categorical variables as sets of numerical variables. Necessary in many types of analysis for them to process categorical data. A common example is using a categorical predictor in regression/ANOVA via dummy coding, effect coding, Helmert coding, user-defined contrasts, etc.

Filter by
Sorted by
Tagged with
0
votes
0answers
6 views

encoding the true labels of radial basis neural network for binary classsification

I am working on a binary classification problem (my class labels are 1 or 0) and I have three layers (input, hidden, output) radial basis neural network. I put two neurons, one per class, in the ...
2
votes
1answer
78 views

Clear explanation of dummy variable trap [duplicate]

I have a confusion in multiple regression about dummy variable trap, so far I had seen tutorials explaining about dummy variable trap and multicollinearity but I'm unable to understand it fully.
4
votes
2answers
18k views

Dummy variable trap?

1) Does including both part time and temporary work cause a dummy variable trap? If not, can we exclude temporary work as an explanatory variable to explain wages in a country as the nature of work is ...
-1
votes
1answer
2k views

Dummy Variable Trap in Stata [on hold]

So basically I am using the wine.dta file and have a range of dummy variables for 6 regions. I have excluded 1 of the regions (regressing on 5 of the dummy ...
1
vote
1answer
386 views

Dummy variable trap in survival models

I am familiar with the dummy variable trap in normal OLS, in which we should include one less dummy variable than the total of categories to avoid the problem of multicollinearity. However, I was ...
1
vote
1answer
80 views

Which ML Algorithms are affected by dummy variable trap?

My understanding is that regression models are affected by the dummy variable trap. What about other machine learning algorithms e.g. linear svm, logistic regression? Also, if an algorithm is not ...
2
votes
1answer
55 views

Setting contrasts for 10-level categorical variable

I have survey data on income and support for environmental protection. Income is a continuous variable that I have broken up into deciles. I have a hypothesis that support for protection ('Agree') ...
0
votes
1answer
18 views

One-hot encoding for duplicate words

I'm currently studying NLP and was practicing one-hot encoding for sentences at the word level. My question is, if we have multiple examples of the same word in a sentence, does one-hot encoding ...
0
votes
0answers
15 views

Interpretation of Marginal Effects for Dummy Variable (using mfx package in R)

So, I calculated a negative binomial regression model and I am trying to estimate the mean marginal effects in R. To do this, I used the mfx package and wrote the ...
1
vote
1answer
21 views

Dummy code interaction among categorical variables

Do you know if there is a way of dummy coding the interactions among three independent categorical variables using SPSS? (with two levels each)
0
votes
1answer
18 views

Dummy coding, ranking of categorical variables

I need to rank categorical variables (top 5 reasons for staying married, top five reasons for divorce). I need to find a method for dummy coding these variables, and then ranking/weighting them (for ...
11
votes
2answers
50k views

Understanding dummy (manual or automated) variable creation in GLM

If a factor variable (e.g. gender with levels M and F) is used in the glm formula, dummy variable(s) are created, and can be found in the glm model summary along with their associated coefficients (e....
1
vote
0answers
26 views

Does the isolation forest care about integer-encoded categorical variables?

The isolation forest (initial paper, follow-up paper) as well as the proposed extended isolation forest (paper) seem like very appealing unsupervised anomaly detection techniques. However, the ...
0
votes
1answer
11 views

How to encode categorical variables in a video game predictive model

I'd like to make a model to predict the result of a match in a video game (win or loss). The game is 3 players against 3 players, and each player has a specific character with specific ...
3
votes
1answer
29 views

Can I remove a dummy variable when it is not significant by itself, but its interaction with another variable is?

I have the following model based on the financial returns of a company as a dependent variable of a stock market index, and a dummy variable interacting with USD exchange rates to my currency. The ...
1
vote
1answer
397 views

Using an overall category as a reference group for dummy variables

I have data on the unemployment rate within 3 education groups for different states, and some other continuous data on for the given states e.g. GDP. I also have the overall unemployment rate for the ...
4
votes
1answer
87 views

How to build a predictive model when more levels of a categorical predictor are possible than appear in the training data

I have created an Artificial Neural Network with 4 categorical features and a binary outcome either 1 for suspicious or 0 for non-suspicious: ...
0
votes
0answers
8 views

Basic question about dummy variables for breakpoint treatment

I am studying basic Econometrics and trying to understand how to deal with breakpoints using dummy variables. I found 3 significant break-points in my data (using 5% confidence) with the Chow ...
2
votes
1answer
46 views

Is it a good practice to drop rare categorical data?

I have a dataset with about 100K samples described mostly by categorical features. The number of unique values in the categories range from 20 to almost 7000. Since these are categorical values and ...
1
vote
1answer
147 views

Encoding of categorical variables with high cardinality

For unsupervised anomaly detection / fraud analytics on credit card data (where I don't have labeled fraudulent cases), there are a lot of variables to consider. The data is of mixed type with ...
0
votes
1answer
496 views

Removing skew from ordinal variables

I'm working on the ames housing data set and wondering how to deal with some string-valued variables. The variable LandSlope can take the values ...
0
votes
1answer
197 views

Does it make sense to apply recursive feature elimination on one-hot encoded features?

Does it make sense to apply recursive feature elimination on a feature set pre-processed with One-Hot Encoding? This is my code for feature selection: ...
4
votes
1answer
105 views

How to Deal with Categorical Variables that Allow Selection of Multiple Values per Observation?

Say you are dealing with a movie database that has movies and their genres. Genre is a categorical variable but each movie can belong to more than one genre. For example, Movie A may be Comedy and ...
0
votes
0answers
16 views

How to call label encoding in multi-label case?

In the multi-label classification, one way to encode the data is to make a table with 1 row for each entry and one column for each label. For each entry/label pair, you get a 0 if the entry has the ...
4
votes
2answers
7k views

Decision Tree - Splitting Factor Variables

I'm new to decision trees and I have some confusion about how factor variables and non-ordered character/string variables get handled in a split. Suppose I have a factor such as "tiny, small, medium, ...
0
votes
0answers
29 views

While performing label encoding or imputation, what should i do to the column with mostly 0's as values which is irrelevant to what column is about?

My DataFrame consists of 2919 rows. Now, for example I have this column "2ndFlrSF" 2ndFlrSF: Second floor's Area in square feet and these are the values in it after I run my Pandas command ...
0
votes
1answer
205 views

Target encoding a categorical variable in a highly imbalanced dataset for binary classification

I have a categorical variable, Industry, that has different values in a dataset that is over 400K datapoints. This dataset is highly imbalanced, the ratio of roughly 99/1. What I am doing is ...
0
votes
0answers
94 views

Target Encoding: missing value imputation before or after encoding

I want to perform a target encoding for my categorical features although I am not sure when to perform the data imputation if any of them has missing values. Let's say I have a few continuous features,...
3
votes
2answers
64 views

Strange encoding for categorical features

I am reading through https://arxiv.org/pdf/1609.06676.pdf which presents an extension of the isolation forest algorithm so that categorical features may be taken into account. On page 5, the authors ...
0
votes
1answer
57 views

PySpark ML: What to do when a logistic regression model is not generalizing?

I created a logistic regression model using PySpark ML. My feature set consists of both categorical and continuous features, and I ran the following to pre-process them: Categorical features: All of ...
1
vote
0answers
272 views

feature embedding for categorical features

I'm training a model and among the features, I have the language of the users. Right now I have done one-hot encoding on the language feature. But I think it would make more sense to have the language ...
1
vote
0answers
118 views

Decision Tree vs Regression for Multiple Categorical Inputs

I have a problem with multiple categorical inputs. These categories do not intuitively map to integers, while preserving their adjacent relationship. Does it make more sense to us a Decision Tree ...
1
vote
1answer
50 views

Linear Regression and High Dimensional Categorical Data

I've read that mean encoding is useful for classification tasks with high dimensional categorical data. My question: What kinds of encodings are effective for high dimensional categorical data in ...
0
votes
1answer
49 views

Encoding variable number of categorical features

I have a dataset listing the software installed for each user. This dataset shall be used (in conjuction with other user datasets) to classify the user into 4 (imbalanced) categories. There are over ...
2
votes
1answer
25 views

Label encoding vs Dummy variable/one hot encoding - correctness?

i understand that when label encoding is used, the numeric number can be interpreted to have an order and a model could assume a linear relationship. However shouldn't this be a problem when there are ...
1
vote
0answers
94 views

When building a model with all categorical features, when do we use dummy variables and when do we use label encoding? [duplicate]

I am working with a dataset that is essentially all categorical data. I have 20-30 distinct columns of categorical data, with some columns having as many as 1000 different categorical values. If I use ...
0
votes
0answers
55 views

How to encode text and categorical variables together?

I have two groups of texts that are very similar (e.g. reviews written on fridays and reviews written on mondays), and I want to build a LSTM that can classify them into positive and negative reviews. ...
1
vote
1answer
180 views

One-Hot Encoding and Feature Engineering While Avoiding Data Leakage

I have a Pandas dataframe for which I've performed some actions over categorical features: Feature Engineering One-Hot Encoding Let's say that in my dataset I have the features "person_income" and "...
2
votes
3answers
188 views

Standardizing dummy variable in multiple linear regression?

I have a multiple linear regression model with several independent variables in different units. Because some of my data is negative, I am unable to take the log and therefore am standardizing the ...
0
votes
0answers
54 views

Should we penalize dummy variables? [duplicate]

Using glmnet we run the following regression cvfit = cv.glmnet(x,y, alpha = 0, intercept = FALSE) where $y$ is the response variable and $x$ is the input matrix....
0
votes
0answers
11 views

Are predictions from an OLS model that only contains categorical covariates biased, if the mean of the residuals does not equal zero?

I understand that the mean of the residuals being zero is a requirement for an OLS model. I also know that when you include the intercept in a regression model, it forces the mean of the residuals to ...
4
votes
2answers
95 views

Variable for logistic regression is categorical and continuous so creates “missingness” in R

I am doing a logistic regression analysis using the glm command in R. It is to identify causes of valve narrowing beyond a certain threshold; 0=no narrowing, 1=narrowed. One of my variables is the ...
4
votes
1answer
58 views

Problems from having too many interactions in a regression?

Excluding the 'dummy variable trap', are the problems from including too many interaction terms in a regression any different from the problems of including too many continuous or binary variables in ...
0
votes
1answer
17 views

Using one-hot encoded features along with continuous-valued features?

The task I wanted to do is a prediction task where most of the features are continuous numbers and some of the features are one-hot encoded. I am training a neural network and I wondered that, is it ...
0
votes
0answers
256 views

Manually setting reference levels in glm of categorical variables [closed]

I have a data set with variables that have 3 or more levels for example pen size: small, medium, large. I know that if you don't set a reference level, R will just pick one and then compare the other ...
1
vote
1answer
40 views

Separate Models vs Flags in the same model

I have customer data from 2 brands. The data structure are the same, but I expected the customer behaviour to be different in different brand. So I could train 2 models, 1 for each brand, or I could ...
0
votes
0answers
43 views

What's the effect of using TF-IDF encoded instead of one-hot encoded categorical data as input to a neural network?

As input into a simple neural network multi-class classifier, I am considering using a variation of the standard one-hot sparse matrix to represent categorical variables. Instead of each element ...
4
votes
1answer
83 views

“Joint” dummy variables for two different variables

I am supposed to show the hazard ratio (HR) stratified by gender (1= female vs. 2= male) and age groups (quartiles, 1-4)*. The combination "female" and "first quartile of age" is supposed to be the ...