Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [categorical-encoding]

Representing categorical variables as sets of numerical variables. Necessary in many types of analysis for them to process categorical data. A common example is using a categorical predictor in regression/ANOVA via dummy coding, effect coding, Helmert coding, user-defined contrasts, etc.

1
vote
0answers
9 views

In the summary of output of the regression analysis in R, is there a way to display categorical variable with just one coefficient

I am doing a linear regression analysis in R with logarithmic dependent variable. One of the control variables is categorical and describes an industry. There are 6 industries and thereby R ...
0
votes
0answers
13 views

Interpreting findings of moderation using PROCESS Macro

I have recently conducted a moderation analysis using the PROCESS Macro plug-in for SPSS. I have tested the moderation effect of W (categorical using dummy variables) on the relationship between X (...
0
votes
0answers
28 views

How do you run a regression when categorical variables may be involved, using R?

I have a data set with several categorical and quantitative variables. Say A, B, and C are categorical with several levels, X and Y are quantitative. I know that X ~ A will basically just be a ...
0
votes
1answer
6 views

Can a dummy variable be used to correct ARCH?

Is it possible to use a dummy variable to allow for a structural break, in order to correct Autoregressive Conditional Heteroscedasticity?
0
votes
0answers
20 views

Linear Regression and High Dimensional Categorical Data

I've read that mean encoding is useful for classification tasks with high dimensional categorical data. My question: What kinds of encodings are effective for high dimensional categorical data in ...
0
votes
0answers
8 views

Dropping One-hot-encoded columns in Pandas/Sklearn

When one-hot-encoding categorial features in python with pandas or sklearn, when should I drop one of the resulting columns? I recall something about having all columns present being a problem for ...
0
votes
0answers
16 views

Finding the fitted probability for an aliased constraint in a binomial model?

For a binomial model, with probit link function: model = glm(response~A+B+C, family = binomial("probit"), na.action = na.omit) where A and B are continuous, C is ...
0
votes
1answer
28 views

Can you use two dummy variables?

Is it possible to use two dummy variables for breakpoints in a linear regression? In EViews I've created the following: ls log(consumption) c log(gdp) log(gdp(-1)) log(consumption(-1)) @year>1989 @...
-1
votes
0answers
15 views

Including all rows for an ID if ID meets a condition once in panel data in R [migrated]

I'm sorry if a similar question has been answered already but I can't seem to find any posts helping me. I wish to define two separate intervention groups (linked to this previous question I asked ...
0
votes
0answers
29 views

Target Encoding: missing value imputation before or after encoding

I want to perform a target encoding for my categorical features although I am not sure when to perform the data imputation if any of them has missing values. Let's say I have a few continuous features,...
2
votes
0answers
8 views

estimation inter rater reliability for strings of characters (i.e. URLs)

I have multiple raters extracting URLs from the internet based on search terms. The core issue is that a URL amounts to a string wherein two raters might come to the same URL but one string is a ...
0
votes
0answers
9 views

Influence of neighbouring characters on a given position

I have 10,000 fixed length strings of DNA sequences. An example would be ATTGGGT M GCGGCTG. Now the character marked M is a position of interest to me (say something that causes diseases). ...
0
votes
1answer
32 views

Multinomial logistic regression reference category function

Bit of background info on the data and methods that I've used. I have a forest road data that consists of one dependent variable which has 4 classes (0-3). The Dependent variable reflects possibility ...
2
votes
1answer
19 views

interpreting coefficients of interaction terms between categorical variables [closed]

Assuming I have a linear regression model of the form: Y = a + (b1*fact1) + (b2*fact2) + b3*(fact1*fact2) + b4*other_var where, fact1, fact2 are categorical variables of 2 factors each, YES and NO ...
0
votes
1answer
11 views

Heterogenous effects in DID models

I am working on a standard DID research design where all the assumptions are met - focused on a policy reform affecting a number of sectors in a country. I would like to improve the design by ...
0
votes
0answers
27 views

Model failed to converge in lme4::glmer() when the a factor is centered or releveled

I'm running a mixed-effects model using glmer() function. The modeling works well with R's default dummy coding. But if I center or relevel a factor of 2 levels, the model failed to converge. I am ...
0
votes
0answers
6 views

Regression analysis incorporating the investment grade status of event country and non-event country

I am conducting a regression analysis estimating the impact of event-country credit ratings on the CDS spreads of non-event countries. As of now the current formula looks like this: 'pct_chg_-1_1 ~ ...
0
votes
1answer
12 views

Manually setting reference levels in glm of categorical variables

I have a data set with variables that have 3 or more levels for example pen size: small, medium, large. I know that if you don't set a reference level, r will just pick one and then compare the other ...
1
vote
1answer
103 views

Is one-hot encoding and standardization of data equivalent to Gower's distance?

For clustering and other techniques for mixed data (numerical and categorical), Gower's distance is usually more preferred than Euclidean distance because the former computes distance differently for ...
0
votes
1answer
36 views

Dummy variables in equation? [closed]

I have a model equation with 10 year and 10 industry dummies. In equation, it will look like this: year$_1$ year$_2$ ... year$_10$ ind$_1$ ind$_2$ .... ind$_{10}$ which is too long to write out. Is ...
3
votes
1answer
82 views

When is deviation coding useful?

After many years of learning about contrasts in linear models I am curious about the relative usefulness of deviation coding, as it is defined by this website. I would appreciate someone filling me in ...
0
votes
1answer
20 views
0
votes
0answers
8 views

Problem with reference category after survSplit in Cox regression

I have two covariates in a survival data, Acet with four factor and Age as a continuous variable. I want category AAA to be the reference in the covariate Acet. Normal Cox regression is fine: ...
0
votes
0answers
14 views

Obtaining the fitted values of main effects and interactions when GLM coding is used

In short, I need to decompose the fitted values obtained from ANOVA into components corresponding to each term in ANOVA statement. This problem appears trivial, but it becomes tricky when GLM coding ...
1
vote
0answers
35 views

Dummy variables with dummyVars() - return to original columns [closed]

For building a machine learning model I used dummyVars() function to create the dummy variables for building a model. ...
0
votes
1answer
42 views

What would be the appropriate statistical test for my study?

What would be the appropriate test for a study that has: 2 categorical independent variables with 2 levels each, 1 moderator variable (and would it matter if my moderator is categorical or continuous?...
1
vote
1answer
18 views

Given two features, one a string and other a categorical, what are the encoding rules?

I have two features in my dataset I'm using to help predict a binary outcome. Based on my features, I'm trying to figure out which I need to drop a dummy to avoid the dummy trap. One feature is a ...
1
vote
0answers
36 views

Dummy variables in Johansen cointegration analysis

I am doing cointegration analysis (all my macroeconomic series are I(1)). In addition, I want to include in all my cointegration analysis a dummy variable indicating the years where there were ...
1
vote
0answers
7 views

Can and should you avoid implicit pretesting for differences in group means?

Suppose I have a population with four (or more) disjoint sub-populations which differ from one another by traitishness, the union of which is the whole population. I have an outcome measure on the ...
0
votes
0answers
15 views

Weird prediction with binary classification on unseen textual data

I try to solve a problem which looks really simple. However I meet an obstacle and get stuck. I have a corpus of texts. I have to assign 0 to 1 to them (appropriate or not). There are a lot of ...
1
vote
3answers
86 views

Multiple regression with dummy variables and interaction term

We have done a multiple regression analysis to see how gender and experience affect salary. We used a dummy variable for gender and then we also added the interaction variable (female work experience)....
0
votes
1answer
16 views

Differences between Wald Statistics and P-values obtained with Dummy Coding vs. Direct Coding in Cox Models

Using the larynx dataset (source: Survival Analysis Techniques for Censored and Truncated Data) to illustrate, supposing you want to estimates the hazard rates of event for stages 2 and stages 3 ...
0
votes
0answers
25 views

How to code a categorical variable for logistic regression with overlap in the categories/subgroups?

Suppose I have a categorical variable consisting of four levels: a, b, c, and d. When these levels are mutual exclusive, I would use dummy coding - so three dummies with for example level a as ...
0
votes
1answer
35 views

One-hot-encoding gives untractable amount of classes

I'm performing regression on the price of bycicles based on their brand, model and submodel. These features are hierarchical: one model belongs only to one brand but one brand can have many models. ...
0
votes
1answer
31 views

Question about making prediction with only two variables

I have a data set with only two variables, student id and book id. I have train and test sets and I will make prediction about what book student will get next time. Should I attach dummy variables to ...
1
vote
1answer
52 views

(Low cardinality) categorical features handling in gradient boosting libraries

In some popular gradient boosting libraries (lgb, catboost), they all seems like can handle categorical inputs by just specifying the column names of the categorical features, and pass it into a ...
0
votes
0answers
26 views

Assess impact on results of participants recruited via additional outreach efforts

I am currently working on a project that evaluates certain study recruitment strategies in regards of increasing participation among different subgroups. One of those strategies is reminder letters. ...
1
vote
4answers
243 views

Regression with Lots of Categorical Variables

I'm facing a regression task with many categorical and few numeric features. I encoded them into dummies and removed the first dummy column for each feature. I am not getting very good R2 at all. I ...
0
votes
1answer
27 views

How do I evaluate/validate my encoding technique?

I have log data and I encode the data for clustering purpose. For example, I have one data column and I represent this unique data in numerical values or binary to be as one column as below. Example ...
0
votes
0answers
55 views

Does it make sense to apply recursive feature elimination on one-hot encoded features?

Does it make sense to apply recursive feature elimination on a feature set pre-processed with One-Hot Encoding? This is my code for feature selection: ...
2
votes
1answer
20 views

How is One-Hot Encoding interpreted by an Algorithm?

I'm new to machine learning, and just learned about the use of one-hot encoding as a method of passing a categorical variable as an input into a machine learning algorithm. As I understand it, one of ...
0
votes
0answers
10 views

Censored Dummy Regressor

I have a dataset that contains factors corresponding income ranges of sampled persons, like people with factor 1 earn between 10,000 to 20,000, 2 between 20,000 to 30,000 . I could just make dummies ...
0
votes
0answers
9 views

How to handle different input sizes of an NN when One-Hot-Encoding a categorical input?

let's assume an input dataset that is a mix of categorical values and real values. When preprocessing this data into an appropriate NN input, OHE is recommended because it doesn't assume any order of ...
0
votes
1answer
30 views

How to handle and test categorical dummy variables when interested only in certain levels?

I want to build a multiple linear regression model. I want to test the effect of a nominal variable with 10+ levels, but I am interested in testing only the effect of 2 of them. 1st Question: How ...
1
vote
1answer
50 views

changing the coding system from helmert coding to difference coding changes regression results?

EDIT: I think I have mistaken the names of the coding systems, so I changed it (in bold). The content has not changed at all, though, so I would still appreciate any answer. END EDIT I'm running ...
0
votes
0answers
28 views

How to encode text and categorical variables together?

I have two groups of texts that are very similar (e.g. reviews written on fridays and reviews written on mondays), and I want to build a LSTM that can classify them into positive and negative reviews. ...
1
vote
1answer
72 views

How to interpret interaction dummies of multiple categories and main effect

I have a panel data crosscountry regression with following structure ($y$ as a drug addiction rate of the country, $x$ as number of homeless of the country and $m$ as HIV infection rate of the country)...
3
votes
3answers
191 views

Can a dummy variable take on more than 2 values?

I am doing a research on foreign direct investment in the EU countries. I came across an article in which the authors assign 4 values to a dummy variable, to be more specific, they assign the value 0 ...
0
votes
0answers
30 views

Encoding variable number of categorical features

I have a dataset listing the software installed for each user. This dataset shall be used (in conjuction with other user datasets) to classify the user into 4 (imbalanced) categories. There are over ...
0
votes
1answer
366 views

How can I one-hot encode a variable that has only 2 levels? [closed]

I'm trying to do OHC in R to convert categorical into numerical data. However R's caret package requires one to use factors with greater than 2 levels. Any idea how to go around this? I've searched ...