Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

(I'm a bit outside my comfort zone, so apologies if this is badly worded, or off-topic)

I have a bibliographic database, containign details of about 1200 different papers, books, web sites etc, all with various details, including keywords and an abstract. I want to somehow analyse this database and produce some graphics showing the correlations between different keywords. (like "drug" is often present with either "pharmacology" or "assay").

Ideally this would be in R, but general advise would also be welcome. (I've seen this question/answer which piqued my interest, and this heatmap graphic also seem related)

My database could be in bibtex, or could be converted to plain text.

share|improve this question
add comment

5 Answers 5

up vote 1 down vote accepted

I'm also outside my area of expertise, but assuming that you want to use R, here are a few thoughts.

  • There is a bibtex package in R for importing bibtex files.
  • Various character functions could be used to extract the key words.
  • The data sounds a little like a two-mode network, which might mean packages like sna and igraph are useful.
  • Plots of 2d multidimensional scaling can also also be useful in visualising similarities (e.g., based on co-occurrence or some other measure) between words (here's a tutorial).
share|improve this answer
add comment

so you have a document x keyword matrix which basically represents a bipartite graph (or two-mode network depending on your cultural background) with edges between documents and tags. If you're not interested in individual documents - as I understand you -, you can create a network of keywords by counting the number of cooccurrences between each keyword. Simply plotting this graph might already give you a neat idea of what this data looks like. You can further tweak the visualization if you, e.g., scale the size of the keywords by the number of total occurrences, or (in case you have a lot of keywords) introduce a minimum number of total occurrences for a keyword to appear in the first place.

As a tool, I can only recommend GraphViz which allows you to specify graphs like

keyword1 -- keyword2
keyword1 -- keyword3
keyword1[label="statistics", fontsize=...]

and "compile" them into pngs, pdfs, whatever, yielding very nice results (particularly if you play a bit with the font settings).

share|improve this answer
add comment

You may want to take a look at the phi coefficient which is a measure of association for nominal variables.

share|improve this answer
add comment

You could try to employ the theory and praxis of association analysis or market basket analysis to your problem (just read "items" as "keywords" / "cited reference" and "market basket" as "journal article").

Disclaimer - this is just an idea, I did not do anything like that myself. Just my 2Cents.

share|improve this answer
add comment

I would recommend using Association Rule Learning for this. It allows you to find words that often co-occur.

If you have a lot of data, it will be much faster than calculating a correlation matrix.

See my video series on text mining here. Includes a tutorial on Association Rules for text.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.