Linked Questions
17 questions linked to/from One-hot-encoding gives untractable amount of classes
1
vote
0
answers
137
views
Finding a latent representation of a high-cardinality one-hot encoded variable [duplicate]
I am working on a clustering project on a dataset that has some numerical variables, and one categorical variable with very high cardinality (~200 values). I was thinking if it is possible to create ...
- 11
111
votes
6
answers
42k
views
Principled way of collapsing categorical variables with many levels?
What techniques are available for collapsing (or pooling) many categories to a few, for the purpose of using them as an input (predictor) in a statistical model?
Consider a variable like college ...
- 12.8k
4
votes
2
answers
20k
views
Difference between linear regression and neural network
I am obviously confused with terms, and different concepts behind it. Each websites gives different intuitions. With all intuitions my brain is full of confusion now. Please help me to address what is ...
- 141
7
votes
2
answers
5k
views
How can I prepare the input layer for recurrent neural network if there are many categorical variables?
I am building a recurrent neural network (RNN). The feature set contains many categorical variables. Some of them are like users and items. In this case, if I use one-hot encoding and concatenate ...
- 2,095
1
vote
2
answers
7k
views
Performing one-hot encoding on a very large dataset
I am currently analysis a data set containing 654281 observations and 27 variables. I aim to perform binary logistic regression and many of my variables are categorical.
I know one hot encoding is ...
- 51
4
votes
2
answers
2k
views
Strange encoding for categorical features
I am reading through https://arxiv.org/pdf/1609.06676.pdf which presents an extension of the isolation forest algorithm so that categorical features may be taken into account. On page 5, the authors ...
- 727
3
votes
2
answers
922
views
How to asses the optimal bag of words vector size?
I have a corpus with 6040592 words and 309074 types (different words). Knowing this information it is possible to know the optimal size of bag of words vectors in order to represent phrases?
I am ...
- 141
3
votes
1
answer
2k
views
Is it a good practice to drop rare categorical data?
I have a dataset with about 100K samples described mostly by categorical features. The number of unique values in the categories range from 20 to almost 7000. Since these are categorical values and ...
- 181
3
votes
1
answer
1k
views
How to decide on encoding high cardinality variables for a small dataset?
I already referred the posts here, here, here, here, here etc. Don't mark as duplicate please.
I have a dataset with 1008 rows with 16 input variables and 1 target variable.
However, 14 of my input ...
- 3,342
1
vote
1
answer
1k
views
Train a RNN with unknown vocabulary size
I'm new to deep learning and i'm trying to code a Visual Question answering network.
I studied and (i think) understood how RNN and LSTM work.
From what i'he understood, i need to train my network ...
1
vote
1
answer
1k
views
feature hashing for high cardinality
when we are applying feature hashing in sklearn it asks us what should be the dimensions of feature required for us. If we decrease too much there will be more collisions which are not good. And we ...
- 31
2
votes
2
answers
380
views
Without encoding, how can we solve high cardinality issue?
I already referred the posts here but this question is different. I don't wish to use categorical encoding. details given below
I have a dataset of 3000 unique customers purchase data. The dataset ...
- 3,342
3
votes
1
answer
435
views
How to interpret feature hashing ouptut?
I am working on a binary classification problem with 1000 rows but have multiple high cardinality variables.
So, I decided to use Hash encoder to avoid curse of dimensionality.
However, after feeding ...
- 3,342
2
votes
2
answers
360
views
One-hot encoding for SOM
I have a question regarding how I should convert categorical data to numerical data. I'm using this kdd99cup intrusion detection dataset, which has a 41 attributes and class label is the type of ...
- 61
0
votes
0
answers
417
views
How to encode categorical data with a lot of unique values and streaming data for anomaly detection
I'm working on a Anomaly Detection problem with streaming data, where i use Robust Random Cut Forest (RRCF). I have 295.000+ rows to start with and there is more data coming in.
The problem is when ...
