Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [categorical-encoding]

Representing categorical variables as sets of numerical variables. Necessary in many types of analysis for them to process categorical data. A common example is using a categorical predictor in regression/ANOVA via dummy coding, effect coding, Helmert coding, user-defined contrasts, etc.

1
vote
1answer
900 views

Do I use dummy encoding or one hot encoding when trying to do regression?

I am trying to do regression for the first time using qualitative and quantitative data using scikit learn. I want to find correlations between user demographic features like age range, country, ...
0
votes
0answers
23 views

Interpretation of logit model without reference category

I have developed a logit model with binary dependent variable in R using the glm function. Overall, the independent variables include seven nominal, 23 ordinal, and two numeric variables. The nominal ...
0
votes
0answers
8 views

What is matrix in year dummies ? Can we regress matrix in year dummy to a dependent variable only to get the residuals? [on hold]

I get confused by this concept I found on journal, It regressing a matrix of year dummy which I do not understand what is it. Does the year matrix dummy means we create a series that represent n-1 ...
1
vote
2answers
184 views

Non-negative matrix factorization (NMF) on mixed data using 1-hot encoding

From a standpoint of interpretation, can I use NMF on one-hot encoded categorical data for dimension reduction? I have mixed data and was thinking about one-hot encoding the categorical features and ...
1
vote
1answer
21 views

Multiple regression with a odd-numbered Likert items

In the questionnaire, individuals were asked to pick from an odd number of options (5 or 7) how much they agree with the statement (from completely agree to completely disagree). Now, if there where ...
0
votes
1answer
19 views

Weird prediction with binary classification on unseen textual data

I try to solve a problem which looks really simple. However I meet an obstacle and get stuck. I have a corpus of texts. I have to assign 0 to 1 to them (appropriate or not). There are a lot of ...
0
votes
0answers
12 views

WoE of test dataset

How to encode the categorical data to Weight of Evidence in the test dataset? I am interested to know about how to handle the categories which are not present in the training set.Or should we combine ...
0
votes
2answers
338 views

dummy vs one-hot encoding - ML for prediction

I understand there is a lack of consensus in the difference (if any) between one-hot (k variables) and dummy (k - 1 variables) encoding from a k-level factor. The caret package seems to auto-encode ...
1
vote
3answers
119 views

Multiple regression with dummy variables and interaction term

We have done a multiple regression analysis to see how gender and experience affect salary. We used a dummy variable for gender and then we also added the interaction variable (female work experience)....
1
vote
1answer
15 views

Transforming categorical variables

Can someone explain me in which cases is a good idea to convert a categorical variable to numerical in order to used it in our model (either regression or classification). I have seen cases where even ...
0
votes
1answer
33 views

Linear Regression in R: factor or z-standardise dichotomous variables

How should one handle dichotomous variables in a linear regression? Is it better to z-standardise or is it better to use factors? For example I have data of an Experiment with a 2x2 Design in which ...
1
vote
0answers
66 views

Dummy variables in Johansen cointegration analysis

I am doing cointegration analysis (all my macroeconomic series are I(1)). In addition, I want to include in all my cointegration analysis a dummy variable indicating the years where there were ...
0
votes
1answer
37 views

General Linear Model applied to binary data

So here I am studying regression analysis. As an assignment, I have been asked to obtain some binary data, simulate its behavior from some sample of it and apply the resulting general linear model to ...
3
votes
1answer
71 views

“Joint” dummy variables for two different variables

I am supposed to show the hazard ratio (HR) stratified by gender (1= female vs. 2= male) and age groups (quartiles, 1-4)*. The combination "female" and "first quartile of age" is supposed to be the ...
2
votes
1answer
23 views

Does dummy code a variable affect the intercept in a linear regression model

My colleague and I were both using R to fit a linear regression with the same dataset and same variables. The outcome variable is test grade while the independent variables are gender, age, and times ...
0
votes
1answer
19 views

h2o glm tweedie for categorical variables

To build a tweedie glm for categorical variables, the document suggested that I can use data['variable_name'].asfactor(). However, in the model output, there is no reference level, i.e., if I have ...
1
vote
1answer
29 views

Separate Models vs Flags in the same model

I have customer data from 2 brands. The data structure are the same, but I expected the customer behaviour to be different in different brand. So I could train 2 models, 1 for each brand, or I could ...
0
votes
0answers
11 views

Should one hot encoding be done on Likert scale ratings?

This seems like an easy question but I haven't been able to find a definitive source for this or questions that address this topic directly. When applying a classification algorithm, should you apply ...
0
votes
0answers
12 views

Interpreting findings of moderation using PROCESS Macro

I have recently conducted a moderation analysis using the PROCESS Macro plug-in for SPSS. I have tested the moderation effect of W (categorical using dummy variables) on the relationship between X (...
0
votes
0answers
20 views

How to deal with a potentially multiple categorical variable

I'm building a model that has, as inputs, some categorical variables. I had already dealt with this sort of data before, and applied different techniques as creation of dummy variables and factor ...
0
votes
3answers
53 views

Does Categorical Variable need normalization/standardization?

since we do normalize as 10kg >>> 10 grams or 1000 >> 10. so incase of one hot encoding eg male=0 and female =1, are we giving more weight to female as 1>0 for training our models?
1
vote
0answers
18 views

Dummy variable reference category

As part of my thesis, I am exploring the effect of firm owner type on research and development, which is a ratio. I have a dummy variable of company owner type with 10 categories e.g. Bank, ...
0
votes
2answers
24 views

Can a dummy variable be used to correct ARCH?

Is it possible to use a dummy variable to allow for a structural break, in order to correct Autoregressive Conditional Heteroscedasticity?
1
vote
0answers
27 views

Is there a theoretical basis for using partial least squares with categorical responses

I am using what is called PLS-DA in JMP to find a model for predicting a categorical (Positive/Negative) response. The documentation says that the responses are simply coded as 0/1, thereby ...
1
vote
1answer
23 views

In the summary of output of the regression analysis in R, is there a way to display categorical variable with just one coefficient [closed]

I am doing a linear regression analysis in R with logarithmic dependent variable. One of the control variables is categorical and describes an industry. There are 6 industries and thereby R ...
3
votes
3answers
336 views

Can a dummy variable take on more than 2 values?

I am doing a research on foreign direct investment in the EU countries. I came across an article in which the authors assign 4 values to a dummy variable, to be more specific, they assign the value 0 ...
0
votes
0answers
38 views

How do you run a regression when categorical variables may be involved, using R?

I have a data set with several categorical and quantitative variables. Say A, B, and C are categorical with several levels, X and Y are quantitative. I know that X ~ A will basically just be a ...
0
votes
0answers
27 views

Linear Regression and High Dimensional Categorical Data

I've read that mean encoding is useful for classification tasks with high dimensional categorical data. My question: What kinds of encodings are effective for high dimensional categorical data in ...
0
votes
0answers
26 views

Dropping One-hot-encoded columns in Pandas/Sklearn

When one-hot-encoding categorial features in python with pandas or sklearn, when should I drop one of the resulting columns? I recall something about having all columns present being a problem for ...
0
votes
0answers
18 views

Finding the fitted probability for an aliased constraint in a binomial model?

For a binomial model, with probit link function: model = glm(response~A+B+C, family = binomial("probit"), na.action = na.omit) where A and B are continuous, C is ...
1
vote
0answers
24 views

Variable selection with tree-structured covariates?

Let's say I want to do regression and that there's a categorical variable which has an inherent tree structure. Using an example from my field of linguistics, let's say I'm trying to predict a binary ...
0
votes
1answer
12 views

Heterogenous effects in DID models

I am working on a standard DID research design where all the assumptions are met - focused on a policy reform affecting a number of sectors in a country. I would like to improve the design by ...
0
votes
1answer
31 views

Can you use two dummy variables?

Is it possible to use two dummy variables for breakpoints in a linear regression? In EViews I've created the following: ls log(consumption) c log(gdp) log(gdp(-1)) log(consumption(-1)) @year>1989 @...
0
votes
1answer
37 views

Multinomial logistic regression reference category function

Bit of background info on the data and methods that I've used. I have a forest road data that consists of one dependent variable which has 4 classes (0-3). The Dependent variable reflects possibility ...
0
votes
1answer
61 views

One-hot encoding for SOM

I have a question regarding how I should convert categorical data to numerical data. I'm using this kdd99cup intrusion detection dataset, which has a 41 attributes and class label is the type of ...
0
votes
0answers
45 views

Target Encoding: missing value imputation before or after encoding

I want to perform a target encoding for my categorical features although I am not sure when to perform the data imputation if any of them has missing values. Let's say I have a few continuous features,...
2
votes
1answer
529 views

Reference level in GLM regression

In GLM regression I have always been told to set the reference level of categorical/ordinal/dummy variables to the level with the most exposure (level with most data), because this somehow makes the ...
1
vote
1answer
279 views

Dummy coding categorical variables with lots of unique values using log2?

I'm trying to understand the logic behind this binary encoder. It automatically takes categorical variables and dummy codes them (similar to one-hot-encoding on sklearn), but reduces the number of ...
2
votes
0answers
9 views

estimation inter rater reliability for strings of characters (i.e. URLs)

I have multiple raters extracting URLs from the internet based on search terms. The core issue is that a URL amounts to a string wherein two raters might come to the same URL but one string is a ...
0
votes
0answers
10 views

Influence of neighbouring characters on a given position

I have 10,000 fixed length strings of DNA sequences. An example would be ATTGGGT M GCGGCTG. Now the character marked M is a position of interest to me (say something that causes diseases). ...
2
votes
1answer
21 views

interpreting coefficients of interaction terms between categorical variables [closed]

Assuming I have a linear regression model of the form: Y = a + (b1*fact1) + (b2*fact2) + b3*(fact1*fact2) + b4*other_var where, fact1, fact2 are categorical variables of 2 factors each, YES and NO ...
0
votes
0answers
28 views

Model failed to converge in lme4::glmer() when the a factor is centered or releveled

I'm running a mixed-effects model using glmer() function. The modeling works well with R's default dummy coding. But if I center or relevel a factor of 2 levels, the model failed to converge. I am ...
0
votes
1answer
50 views

Manually setting reference levels in glm of categorical variables

I have a data set with variables that have 3 or more levels for example pen size: small, medium, large. I know that if you don't set a reference level, r will just pick one and then compare the other ...
0
votes
0answers
8 views

Regression analysis incorporating the investment grade status of event country and non-event country

I am conducting a regression analysis estimating the impact of event-country credit ratings on the CDS spreads of non-event countries. As of now the current formula looks like this: 'pct_chg_-1_1 ~ ...
2
votes
1answer
139 views

Is one-hot encoding and standardization of data equivalent to Gower's distance?

For clustering and other techniques for mixed data (numerical and categorical), Gower's distance is usually more preferred than Euclidean distance because the former computes distance differently for ...
3
votes
1answer
955 views

How to handle too many categorical features with too many categories for XGBoost?

In my data I have 35 features and 14 of them are categorical. Half of them have 3 to 4 categories but others have 14 to 28 categories. One Hot Encoding them would only lead to a sparse matrix with ...
0
votes
1answer
37 views

Dummy variables in equation? [closed]

I have a model equation with 10 year and 10 industry dummies. In equation, it will look like this: year$_1$ year$_2$ ... year$_10$ ind$_1$ ind$_2$ .... ind$_{10}$ which is too long to write out. Is ...
4
votes
1answer
124 views

When is deviation coding useful?

After many years of learning about contrasts in linear models I am curious about the relative usefulness of deviation coding, as it is defined by this website. I would appreciate someone filling me in ...
2
votes
1answer
7k views

Binary Encoding vs One hot Encoding

what is the difference between binary Encoding and one-hot for categorical input variables for English Text and their impact on the neural network ? can anyone help me to find a scientific paper about ...