I have a dataset I'm using to predict a binary outcome variable with 6 columns. Five of them are 10-30 level categorical variables with information about the user, e.g. job function, industry, country, but one is a raw, user entered text field (job title) with thousands of discrete values. In order to perform the prediction, I need to reduce Job Title to a usable number of categories (say, 100 or less).

Do any standard or published approaches exist for single vector dimension reduction? I haven't been able to find anything online.

My current approach is as follows. I've looked at PCA, but it doesn't seem to be applicable to this problem, as I'm only interested in reducing the single vector. Instead what I've done is group some job titles by generating a matrix containing the Levenshtein distance between each string and grouping the best matches. However, this mostly only serves to group mis-spellings, and I worry about losing important information (e.g. grouping SVP with VP). The second step is to simply select the 100 most frequent character strings and push the rest to "Uncommon Title", which I treat as it's own category in prediction. The problem is that after both of these steps, I'm left with roughly 50% of my observations as having the "Uncommon Title" category. Is there anything I can do to maintain at least some information on these observations?


First, I would make all of the inputs to this text string either upper or lower case. Sounds obvious but many people aren't aware that many software modules evaluate strings differently as a function of case. Then, I would suggest exploring a few text functions, e.g., Hamming's distance, Levenshtein distance, or hash operators that can make the collapsing and condensing of text strings more manageable by quantifying the amount of effort in terms of character substitution or transposition required to make the strings equivalent. Obviously, focusing on the strings with low values for these functions would help reduce the workload. Finally, consider biting the bullet and grooming the remaining data by hand for typos, variant spellings, abbreviations, punctuation, and so on may be a necessary last step -- but only on the strings that are the most difficult, hopefully a subset of the total. E.g., not being sure what you've done to get down to 50% "Uncommon" strings, you might focus on those fields.

  • $\begingroup$ Thanks for the feedback, it's good to know I'm more or less on the right track. I had pushed everything to lower case as you mentioned, but of course you're correct that it's a very necessary first step. I think if I were to try and take a programmatic approach, I think it would seem that a function where the acceptable distance increases as the relative frequency decreases would be the appropriate term. That can at least help you create relatively even coverage across each category within the vector. $\endgroup$ – Stencil Jun 11 '15 at 0:58
  • $\begingroup$ That's true and a good workaround doing it by frequency and proximity. I hate to admit it, but having been given any number of dirty fields like this over the years, I've usually ended up brute forcing it, i.e., grooming the d**n things by hand. Even with thousands of possible values, one person can spend a day or so (depending). It's only when the unique counts get into the tens of thousands that you have to call out the cavalry. If it's really huge and ugly, browse "Amazon Turks filetype:pdf" for papers on crowdsourcing the dirty work. $\endgroup$ – Mike Hunter Jun 11 '15 at 1:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.