I have a column X which contains occupation/profession as an independent variable as free text, which is very much correlated with a continuous dependent variable.

What techniques do you usually use to 'bin' such free text variables for regression?

I am thinking about breaking the free text strings down into quadgramms (4 characters) and thus create a high dimensional matrix. I could then apply word embedding/SVD etc. to create a low-ish dimensional matrix. The matrix could then be clustered and the cluster membership would represent the binned value.


The problem is that the similarity between two occupations is not measured well by the similarity between the words for them: doctor is to surgeon as project manager is to Scrum leader, but you can't tell that from the character strings alone.

If you can find an ontology of professions, that would be ideal. Insurance companies use such things, but I don't know whether there is a free one.


Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.