Suppose we have a dataset $X$ of features and a target binary prediction $y\in\{0,1\}$ for each datapoint. Each row of $X$ consists of counts (a bag of features). We can normalize each row of $X$ to get proportions of each feature.

As a concrete example, imagine that each row consists of a bag of words the users received from messages on a social media platform and $y$ is whether or not the user returned to the platform at a future date. I'd like to figure out which features are important for predicting $y$, and/or train a reasonably good classifier. Unfortunately, $X$ here is highly sparse (zero-inflation).

One important subset of features can consist of "toxic words", which do heavily influence $y$ (the more you get, the less likely you are to return). On the other hand, not getting any toxic words is also bad, because obviously getting less messages overall would lead to lower activity in the future. So across the proportion of toxic words received, the mean value of $y$ is essentially hill-shaped, going down if you got little-or-no toxic words, and going down if you received too many. If we manually select a keyword list of toxic words, they will occur in around 1% of messages. In other words, I can measurably see a negative impact of having a higher proportion of toxic words, but there's also the negative impact of having none.

I'm wondering what approaches are out there for training a classifier for such a problem. Note that I'm willing to use embeddings (e.g Word2Vec) to de-sparisfy the above, but the fact that there's a strong zero-inflation, which negatively correlates with $y$, makes me wonder what the correct loss function is. Surely this is a well-known problem in statistics dealing with zero-inflation, so any references would be greatly appreciated!

It seems like I need to do some kind of negative sampling on words, while simultaneously picking a minimum threshold for the frequency of the word for each user, in a given training batch?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.