I have a text field dataset. Each observation counts the number of appearances of that particular word, and the columns (variables) are most frequently appeared words. Within each column, zeros dominates with a percentage of over 90%. I have a binary response variable that I am predicting. So far, CART and neural network methods failed, and they always predict way too many 1s than 0s, and sometimes even don't predict any zeros. What methods/other models should I use to improve the accuracy?
$\begingroup$
$\endgroup$
2
-
3$\begingroup$ Niether CART not Neural Networks predict ones or zeros, they predict probabilities. $\endgroup$– Matthew DruryAug 3, 2017 at 22:54
-
2$\begingroup$ There's not an easy answer to this, but I'm compelled to suggest 'feature engineering'. You have too many sparse features. Suppose that instead of treating the words 'like','love' and 'admire' separately, you combined them into a single feature called 'positive'? A difficult task, but often makes the difference. $\endgroup$– HEITZAug 4, 2017 at 3:43
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
I would propose a few potential issues with your current procedure. (2) is more likely the primary issue.
Feature engineering / model selection: It may be the case that the models or the features are not well suited to extracting your signal of interest. This question is an empirical one, you should consider if your data can reasonably perform the task. You might also consider dimensionality reduction techniques (word2vec springs to mind) once you have constructed your features, but I would first try and see if you can get decent performance with a lasso using TFIDF features and build out from there.
You are, in general, going to get a probability from these classification models. You can adjust the threshold (between 0 and 1) to maximize your performance metric of interest. Sometimes, implementations of these algorithms have a default threshold of 0.5 (everything above becomes a 1, everything below a 0). You don't have to use the 0.5 threshold, you can adjust it to maximize your precision subject to some minimum recall, for example. Sounds you like should try adjusting it upward in your case.
Class imbalance: If you have many more examples of one class than another, you may end up with a model that emphasizes performance on one class (for example, if you have 99% positive examples, a model can get 99% accuracy by just classifying everything as positive). If you have class imbalance, consider up or down sampling or weighting upwards of observations of the minority class.
-
$\begingroup$ I have similar issue here. My dataset has 74% of 0s and 1s! I removed them and my accuracy got better using LSTM, but I am worried about credibility of my results. What do you think please about this? Should I use dimensionality reduction or it's nor recommended based on my dataset? $\endgroup$– AvvJul 9, 2021 at 14:19
$\begingroup$
$\endgroup$
3
I use four methods in case of excesses of zero:
- Double Hurdle (the first step is a logit or probit)
- Zero Inflated (for discrete distribution as your case would be a ZIB Zero Inflated Binomial) - you could add an EM algorithm to improve accuracy.
- Tweedie Distribution (with a case that in my opinion is not applicable to your problem)
- Bayesian -> MCMC
-
$\begingroup$ Thank you, so what about deep learning, how it should handle dataset with lots of 0s in case I am building an autoencoder please? $\endgroup$– AvvJul 9, 2021 at 14:29
-
$\begingroup$ Specifically, NN doesn't need to implement the Tweedie distribution to solve this problem (I'm asking if zero inflation is a problem for NN, IMO isn't). So is NN strictly necessary for your research question, or can it be replaced with other machine learning tools? $\endgroup$ Jul 10, 2021 at 15:09
-
$\begingroup$ thanks for replying back. I just need to have good embeddings for my timeseries input, so I think NN is a good hit. But my dataset has lots of 1s and 0s, around 70%. $\endgroup$– AvvJul 10, 2021 at 19:07