Setting and Data
I would like to run a 2-stage Hurdle regression with various variables describing the funding activity of companies (number of rounds, amount, etc). Some information on the data set:
- 60,000+ rows (i.e. companies)
- 20+ variables (numerical, integer, categorical, binary)
I have the feeling that one key control variable of my regression would be the industry of the company.
Unfortunately such information is not readily available but rather the variable named "company category" that consists of ~850 different category tags (e.g., 3D printing, greentech, fintech, etc). Each company has n assigned category tags that collectively shall describe the type of business they are running (~94% of companies have 5 or less tags - min is 1; max is 48). Unfortunately the category tags are not reported in the order as the author assigned them, but alphabetically.
Exemplary data - alternatively also in long format
company_name company_category_list (chr) (chr) 1 Jurnal.id Accounting|Software 2 Magicpin Retail 3 MoneyMe Financial Services 4 ScoopWhoop Publishing 5 Stride Travel Adventure Travel|Online Travel|Reviews and Recommendations|Search
What I am trying to achieve
To be able to include this information in a meaningful way into my regression as a categorical variable I need one element/value per company (i.e. row)
-> Is this assumption true? Or is it actually possible to include a variable with multiple categorical values per observation in a regression without having to create 850 dummy variables (one for each category tag)?
To generate one element per company I thought of identifying industry clusters.
Below is a snapshot of the company/category tag information converted into a binary matrix (rownames are the company IDs). The real matrix only contains ~0.3% of 1s - the rest are 0s (i.e., highly sparse matrix)
3d 3d printing 3d technology accounting active lifestyle [...] 1 0 0 0 1 0 2 1 0 0 0 0 3 1 1 0 0 0 4 0 0 1 0 0 5 0 0 0 0 1 [...]
I tried so far the following clustering algorithms (based on suggestions in other threads):
- DBSCAN -> seems not applicable as not really suitable for sparse, binary data
- ROCK -> despite running it on a small server (128 GB RAM, 12 cores) R session gets terminated when running rockLink within rockCluster (after computing distances)
- k-modes: did not return any meaningful clusters
What I tried besides clustering
Reducing the number of category tags by similarity/frequent co-occurance (based on cramer's v), string distance and tf idf. This reduced the number of tags to around 600 -> still way too high
Treating the category tags collectively per company as a sentence, splitting everything into single words, erasing stop words and reducing then the number of words in the same way as above - wasn't much of an improvement compared to bullet (1)
As mentioned above, is clustering actually the way to go or are their other ways to reach my goal of including the category data into my regression?
If clustering is the way to go, do you have any concrete recommendations which approachs/algorithms I should try considering my input data (binary/categorical, very large)?