I have a bunch of job descriptions entered by users. There are all sort of misspells and bad data. i.e:

tulane univ hospital
tullett prebon
weik investment
weill cornell university medical center
weiss waldee hohimer dds
welded constrction l.p.

What steps would you take to 'augment' this values with job related insights ?

The best I can think of is to give it to wolfram alpha. But I wonder if there are other accessible techniques that I can utilize using python.

Update: I found out that there is a Standard Occupational Classification, I really would like to match the name to the SOC and the SOC to a range of average salaries.

  • $\begingroup$ Can you map these manually? $\endgroup$
    – Aksakal
    Dec 5, 2014 at 2:27
  • $\begingroup$ No, there are 10,000+ of work descriptions .. $\endgroup$
    – fabrizioM
    Dec 6, 2014 at 1:00
  • 2
    $\begingroup$ 10,000 words is about 22 pages. Not that much if you ask me. $\endgroup$
    – Aksakal
    Dec 6, 2014 at 1:48

3 Answers 3


A potential way to start this is to make use of Python's Natural Language Tool Kit (NLTK) which can be utilized for text and topic analysis but also has useful functions to extract certain words from strings. For instance, you could extract from the job description the words "medical", "hospital", etc. in order to find broad occupations and sectors. Due to the spelling mistakes and quality of the data I don't think it can be done in a fully automated fashion such that you might end up coding the SOCs yourself. Nonetheless, having the broad occupations and sectors in this way already makes the task a lot easier.

If you are interested in natural language processing/text and topic analysis/text mining beyond this, a fairly inexpensive but useful book is by Bird et al. (2009) "Natural Language Processing with Python".

Occupational titles have been linked to salaries by David Autor. He linked data in the Current Population Survey (the data which is used to also produce U.S. unemployment figures) to the SOC titles from which you can also get salaries in each occupation. From these you can easily compute mean salaries in each occupation and you can even have an idea about the variance (within occupational earnings inequality) in each occupation. David makes his data sets available on his data archive at MIT.


I've had success using Latent Dirichlet Allocation (LDA) to find the latent themes or "topics" in textual data. LDA will create $k$ topics out of terms (words) from your corpus of job descriptions. Each job description is given a probability of containing each of the $k$ topics. For example if you asked LDA to classify a corpus into 3 topics, a job description for a graphic designer might have 80% "photoshop graphic illustrator...", 18% "HTML CSS JS...", and 2% "Java Spring object-oriented...". There's plenty to read about the LDA, just search or start with the Quora question.

My analysis with LDA was in R but there is of course a Python package although I have never employed it in my own work.

You might consider selecting a topic number that corresponds with the number occupations in the SOC. Once you have generated the topics inspect them and see if you can find meaningful links to the SOC and adjust the topic number accordingly until you are satisfied.

To make salary estimates for each job description consider weighting each salary using the topic probabilities. For example if a job description had an 80% probability of being a software developer SOC weight the salary by .80 and the remaining topics likewise. If that creates too much noise just set a cutoff (maybe 20%) and remove the remaining topic weights from the salary estimate.

For misspellings you can always attack it with a spell checker and see how it compares to the results without the tool. Also make sure to employ standard NLP techniques such as punctuation removal and word stemming prior to running LDA.

  • $\begingroup$ should be relatively easy to do spell checking type procedure.( or rather creating a list of all the words used with word count and use that (eg looking at rarest words only) to remap to correct spelling. might want to look at OpenRegine github project (formerly google open refine) for the tidy up $\endgroup$
    – seanv507
    Dec 11, 2014 at 1:36

Those are not so much job descriptions as job titles. If you did have descriptions like this example from the SOC definitions, you could use a topic model as suggested by Chris:

1011 Chief Executives Determine and formulate policies and provide overall direction of companies or private and public sector organizations within guidelines set up by a board of directors or similar governing body. Plan, direct, or coordinate operational activities at the highest level of management with the help of subordinate executives and staff managers.

In the absence of long-form text, you could use a naive Bayesian classifier (since you have a classification problem) that uses the social network as a feature, since people are likely to work in the same types of jobs as their friends. Another feature could be the string similarity to the Direct Match Title File (I think this database is just what you need), which provides a mapping between job titles and the SOC.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.