How would you categorize / extract information out of job descriptions?

Question

I have a bunch of job descriptions entered by users. There are all sort of misspells and bad data. i.e:

...
tulane univ hospital
tulip
tullett prebon
... 
weik investment
weill cornell university medical center
weis
weiss waldee hohimer dds
welded constrction l.p.
welder
welder
welder
...

What steps would you take to 'augment' this values with job related insights ?

The best I can think of is to give it to wolfram alpha. But I wonder if there are other accessible techniques that I can utilize using python.

Update: I found out that there is a Standard Occupational Classification, I really would like to match the name to the SOC and the SOC to a range of average salaries.

Answer 1

A potential way to start this is to make use of Python's Natural Language Tool Kit (NLTK) which can be utilized for text and topic analysis but also has useful functions to extract certain words from strings. For instance, you could extract from the job description the words "medical", "hospital", etc. in order to find broad occupations and sectors. Due to the spelling mistakes and quality of the data I don't think it can be done in a fully automated fashion such that you might end up coding the SOCs yourself. Nonetheless, having the broad occupations and sectors in this way already makes the task a lot easier.

If you are interested in natural language processing/text and topic analysis/text mining beyond this, a fairly inexpensive but useful book is by Bird et al. (2009) "Natural Language Processing with Python".

Occupational titles have been linked to salaries by David Autor. He linked data in the Current Population Survey (the data which is used to also produce U.S. unemployment figures) to the SOC titles from which you can also get salaries in each occupation. From these you can easily compute mean salaries in each occupation and you can even have an idea about the variance (within occupational earnings inequality) in each occupation. David makes his data sets available on his data archive at MIT.

Answer 2

I've had success using Latent Dirichlet Allocation (LDA) to find the latent themes or "topics" in textual data. LDA will create $k$ topics out of terms (words) from your corpus of job descriptions. Each job description is given a probability of containing each of the $k$ topics. For example if you asked LDA to classify a corpus into 3 topics, a job description for a graphic designer might have 80% "photoshop graphic illustrator...", 18% "HTML CSS JS...", and 2% "Java Spring object-oriented...". There's plenty to read about the LDA, just search or start with the Quora question.

My analysis with LDA was in R but there is of course a Python package although I have never employed it in my own work.

You might consider selecting a topic number that corresponds with the number occupations in the SOC. Once you have generated the topics inspect them and see if you can find meaningful links to the SOC and adjust the topic number accordingly until you are satisfied.

To make salary estimates for each job description consider weighting each salary using the topic probabilities. For example if a job description had an 80% probability of being a software developer SOC weight the salary by .80 and the remaining topics likewise. If that creates too much noise just set a cutoff (maybe 20%) and remove the remaining topic weights from the salary estimate.

For misspellings you can always attack it with a spell checker and see how it compares to the results without the tool. Also make sure to employ standard NLP techniques such as punctuation removal and word stemming prior to running LDA.

Answer 3

Those are not so much job descriptions as job titles. If you did have descriptions like this example from the SOC definitions, you could use a topic model as suggested by Chris:

1011 Chief Executives Determine and formulate policies and provide overall direction of companies or private and public sector organizations within guidelines set up by a board of directors or similar governing body. Plan, direct, or coordinate operational activities at the highest level of management with the help of subordinate executives and staff managers.

In the absence of long-form text, you could use a naive Bayesian classifier (since you have a classification problem) that uses the social network as a feature, since people are likely to work in the same types of jobs as their friends. Another feature could be the string similarity to the Direct Match Title File (I think this database is just what you need), which provides a mapping between job titles and the SOC.