Applying TF/IDF to non-text data? I have a classification problem in which I am supposed to predict the end state of an object based on a set of events it experiences. There are about one thousand possible events and each object is impacted by 10-50 events.
First, I tried logistic regression (using glmnet to be specific) and got AUC = 0.59. My plan was to try xgboost next, but when I looked the data again, I realised that it looks similar to bag-of-words, with a lot of zeros and small integers here and there. That made me want to try TF-IDF for feature engineering: the events are words, and the objects are documents. Right away I got AUC > 0.7.
I did some googling to find more information about this approach, but could not find anything. So, my questions are:


*

*Can TF-IDF be used for non-text data?

*what are some potential pitfalls of this approach?

*Are there any links or references to existing applications of TF-IDF for non-text data?

 A: 
  
*
  
*Can TF-IDF be used for non-text data?
  

Sure. I've used it very successfully on medical records where every ICD-10 diagnosis or procedure code is a "word" and every patient is a "document." Very common procedures like EKGs and lipid panels get lower weights. This allows LSA to be applied. While we ultimately went with an LDA model which doesn't need TF-IDF preprocessing, the LSA model with TF-IDF definitely worked as intended.


  
*what are some potential pitfalls of this approach?
  

Many models, such as linear or logistic regression, ignore the scale of individual features; for those models, the "IDF" part isn't necessarily doing any work. However, if your using a regularized model such as ElasticNet, then the scale becomes important. Usually we want to standardize each feature by centering and scaling, but TF-IDF can also be used as a principle way to assign different scales to each feature. This brings us to a further complication: TF-IDF isn't one concrete formula like MSE. If you say MSE, I could write down the equation, but there are lots of variations of TF-IDF. Wikipedia lists several variations, for example.
It's rarely possible to guess ahead of time which scaling strategy will work best. I recommend using cross validation to compare many scaling strategies, such as raw TF, centered and scaled, and several TF-IDF variants. 


  
*Are there any links or references to existing applications of TF-IDF for non-text data?
  

In addition to my own project, I've seen other papers that use TF-IDF on medical codes. I'm not sure where else its been used, but it's potentially useful anywhere you can make a clear analogy for "words" and "documents."
A: I’ve used TF-IDF and LDA on non-text, and am not aware of pitfalls in general. That doesn’t mean there aren’t any pitfalls, though! And, of course, any bag-of-words approach is throwing away your time series’ ordering, which can dramatically affect meaning.
In your use case, there are a couple of thoughts that come to mind:


*

*It seems that there are no amounts associated with your events? In textual use, you tend to have word counts within documents. So you are working with just presence/absence data, which is even less information from the original time series.

*Your solution appeals to me, and you can make the analogy that each object is a “document” and the events that happened to it are “words” and you’re looking to categorize your objects (“documents”) by “topic”.  But some folks won’t accept that and will demand a technique they are familiar with, instead of an off-brand usage.

*I’d investigate your logistic regression more. I’ve heard of successful logistic regressions with 10,000 variables. They used Feature Hashing to compress the huge number of highly sparse features down to 10,000.

*Have you tried an association rules approach? Do certain bigrams (adjacent pair of events) have high association with your target output? Be careful, though, since slicing your data more and more finely can cause associations “by chance” — look at enough cases, especially with small numbers, and you will eventually find something.
