All Questions
Tagged with categorical-encoding python
32 questions
0
votes
0
answers
12
views
Question about the effects of cardinality changing on the model's training
I'm analyzing a dataset that contains a feature 'street_name' with 5980 unique values. I used the LeaveOneOutEncoder class for encoding, but I noticed that the cardinality reduced a lot. There are now ...
2
votes
1
answer
39
views
Regression predictor from count of categorical variables?
Let's say I have the following strings and associated target variables:
...
- 173
0
votes
1
answer
69
views
OLS model specification that includes all dummy variables with a predetermined coefficient
I'm working with a OLS model that includes dummy variables (quarters of year). Here's what I would specify it:
$$y = \beta X + \gamma_1Q_1 + \gamma_2Q_2 + \gamma_3Q_3 + \epsilon$$
However, in the ...
- 769
3
votes
1
answer
527
views
Measure the distribution shift of categorical variables
In the model performance monitoring stage, if you want to measure the change of a distribution of a numerical variable, Population Stability Index can be used. But ...
- 151
1
vote
1
answer
321
views
Encode multiple values of an attributes in Pandas
I have a dataset and one of the attributes of the dataset is Race. People have multiple races on the dataset. The values for the attribute Race are following
...
- 175
5
votes
3
answers
1k
views
why use dummies and not different integers for categorical data? [duplicate]
I am new to stats and I want to use a regression to determine income
See below table example
age
class
location
income
23
Adult
London
23000
44
Adult
Glasgow
45000
75
Senior
Birmingham
37000
12
...
- 51
0
votes
0
answers
241
views
Framework for applying weights to binary variables in regression
Say I am training a ridge regression model on nothing but binary variables. The context being that each variable represents a player - a value of 1 meaning they were playing the game at the time, ...
- 89
2
votes
1
answer
152
views
Is it important to convert numeric features to object if they have no ordinal or mathematical meaning?
For example a column containing numeric values for phone area code or a postal code.
In case it matters, I am preprocessing data for use in a tree-based ensemble classifier.
- 29
0
votes
0
answers
49
views
How to decide n_componets in HashEncoding
I want to predict salary for each person depending on the job_roles, Technical skills. Column job_roles have 1500 unique roles.
There is a column Technical skills (which have combination of skills) I ...
- 63
0
votes
1
answer
56
views
It is possible to use regression to measure the correlation between a continuous variable and a dummy variable?
I have 3 columns: one column is a continuous variable (e.g., age) and the other two columns represent dummy variables with values 1 and 0 (yes/no).
What kind of regression do I have to use to measure ...
- 7
3
votes
1
answer
2k
views
What dummy variables to set for US state-and-year Differences-in-Differences model
As a sort of personal experiment, I'm trying to run a differences-in-differences (DiD) model on US state-level firearms restrictions and violent crime, to see if changes in the former impact the ...
- 33
0
votes
0
answers
252
views
Label-encoding nominal variables
I am aware of the practice that label encoding is preferred for ordinal variables while ...
- 171
1
vote
0
answers
54
views
Modeling with Multiple Values per Variable, per Observation
I'm attempting to develop an autoencoder on top of medical claims data that have mutliple values per category because a claim often has multiple lines associated with it. For example, let's say (but ...
- 227
1
vote
0
answers
282
views
OneHotEncoding and Scaler in Pipeline, avoid data leak?
so I have my data and split it in the beginning in test and train set.
Then I apply following Pipelines on it:
...
- 121
2
votes
1
answer
2k
views
Can one do one-hot encoding with Count Vectorizer?
I am new to machine learning. I am stuck with a doubt about whether we could do one-hot encoding using bag of words (e.g. scikit-learn's CountVectorizer)?
- 29
0
votes
1
answer
919
views
Dealing with over 1000 categorical values (which are also a unique identifiers)
I am preparing my dataset for a logistic regression and need to check how best to handle a column with categorical values. As the dataset is for sales transactions, the column in question is the ...
- 1
1
vote
0
answers
162
views
Interpretation of categorical variable coefficients in Linear Regression using sklearn's OneHotEncoder
I'm getting a little confused: when it comes to explaining the interpretation of coefficients of dummy variables in ML, all the sources say that one category is the reference level (i.e. 0) while the ...
- 11
0
votes
1
answer
183
views
In a Dataset with many categorical columns what should be preferred -One Hot encoding or Label encoding when doing regression?
Suppose my Dataset for automobiles has a feature 'Number of cylinders' with labels 'One','Two'..(Strings) as categories,what should be preferred label encoding or One hot encoder?
- 25
0
votes
2
answers
4k
views
What is the best way to handle ordinal features having numeric values in python? [closed]
What is the best way to encode ordinal feature?
Is it by transforming it using OneHotEncoder so values going from 1 to 7 lets say would become head of new field feature.
Or by using StandardScaler() ...
- 153
0
votes
0
answers
1k
views
Conditional linear regression with indicator variables (Python)
I have the sample dataset below 20 observations of Y variables and 20 observations of X variables. Both are normalized (z-scored).
I have a prior that (i) larger magnitude X values with $abs(X) >= ...
- 101
1
vote
1
answer
68
views
How to encode categorical variables in a video game predictive model
I'd like to make a model to predict the result of a match in a video game (win or loss).
The game is 3 players against 3 players, and each player has a specific character with specific ...
- 483
0
votes
1
answer
176
views
Using ANOVA to judge Yes/Sometimes/No questions ability to significantly predict a continuous variable?
I have a dataset that looks something like this:
...
- 101
2
votes
1
answer
3k
views
Does it make sense to apply recursive feature elimination on one-hot encoded features?
Does it make sense to apply recursive feature elimination on a feature set pre-processed with One-Hot Encoding?
This is my code for feature selection:
...
- 21
4
votes
1
answer
3k
views
VIF Drops Significantly When I Delete Some Dummy Variables
Is my model valid even with the high VIF? Does it matter which dummy variable I drop as the reference point?
I have a a category variable (Fruit) that I converted ...
- 205
1
vote
1
answer
2k
views
2
votes
1
answer
699
views
One-Hot Encoding and Feature Engineering While Avoiding Data Leakage
I have a Pandas dataframe for which I've performed some actions over categorical features:
Feature Engineering
One-Hot Encoding
Let's say that in my dataset I have the features "person_income" and "...
- 121
2
votes
1
answer
117
views
How to encode noisy text data to make it suitable for supervised machine lea
I want to create an ML program that cleans up noisy data. I have the raw text features and the labels. ML programs tend to prefer numeric data, so I need to encode my text features. This is an example ...
- 131
-1
votes
3
answers
592
views
Dealing with categorical variables - Looking for recommendations [closed]
I have the following dataset, in which the wind direction (Direccion del viento (Pos)) column is categorical, with 8 categorical values:
In total Direccion del ...
- 99
1
vote
1
answer
3k
views
Reverse a label encoded target in test and train series?
when performing a Scikit train/test split like so:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
with categorical ...
- 49
13
votes
4
answers
6k
views
How to statistically prove if a column has categorical data or not, using Python
I have a data frame in python where I need to find all categorical variables. Checking the type of the column doesn't always work because int type can also be ...
- 283
1
vote
0
answers
25
views
Regression Modeling with Secondary Categorical Values and Missingness
I am having a hard time making a decision with how to handle missing data under a specific set of circumstances and what it means to the model.
Consider that I have the following fictitious dataset.
...
- 113
1
vote
1
answer
3k
views
Do I use dummy encoding or one hot encoding when trying to do regression?
I am trying to do regression for the first time using qualitative and quantitative data using scikit learn.
I want to find correlations between user demographic features like age range, country, ...
- 219