I have a text field dataset. Each observation counts the number of appearances of that particular word, and the columns (variables) are most frequently appeared words. Within each column, zeros dominates with a percentage of over 90%. I have a binary response variable that I am predicting. So far, CART and neural network methods failed, and they always predict way too many 1s than 0s, and sometimes even don't predict any zeros. What methods/other models should I use to improve the accuracy?
I would propose a few potential issues with your current procedure. (2) is more likely the primary issue.
Feature engineering / model selection: It may be the case that the models or the features are not well suited to extracting your signal of interest. This question is an empirical one, you should consider if your data can reasonably perform the task. You might also consider dimensionality reduction techniques (word2vec springs to mind) once you have constructed your features, but I would first try and see if you can get decent performance with a lasso using TFIDF features and build out from there.
You are, in general, going to get a probability from these classification models. You can adjust the threshold (between 0 and 1) to maximize your performance metric of interest. Sometimes, implementations of these algorithms have a default threshold of 0.5 (everything above becomes a 1, everything below a 0). You don't have to use the 0.5 threshold, you can adjust it to maximize your precision subject to some minimum recall, for example. Sounds you like should try adjusting it upward in your case.
Class imbalance: If you have many more examples of one class than another, you may end up with a model that emphasizes performance on one class (for example, if you have 99% positive examples, a model can get 99% accuracy by just classifying everything as positive). If you have class imbalance, consider up or down sampling or weighting upwards of observations of the minority class.
I use four methods in case of excesses of zero:
- Double Hurdle (the first step is a logit or probit)
- Zero Inflated (for discrete distribution as your case would be a ZIB Zero Inflated Binomial) - you could add an EM algorithm to improve accuracy.
- Tweedie Distribution (with a case that in my opinion is not applicable to your problem)
- Bayesian -> MCMC