8
$\begingroup$

(I'm a bit outside my comfort zone, so apologies if this is badly worded, or off-topic)

I have a bibliographic database, containign details of about 1200 different papers, books, web sites etc, all with various details, including keywords and an abstract. I want to somehow analyse this database and produce some graphics showing the correlations between different keywords. (like "drug" is often present with either "pharmacology" or "assay").

Ideally this would be in R, but general advise would also be welcome. (I've seen this question/answer which piqued my interest, and this heatmap graphic also seem related)

My database could be in bibtex, or could be converted to plain text.

Improve this question
$\endgroup$
1
  • $\begingroup$ did you finally find an interesting solution for this problem? $\endgroup$
    – Open the way
    Oct 26, 2016 at 13:31

5 Answers 5

Reset to default
1
$\begingroup$

I'm also outside my area of expertise, but assuming that you want to use R, here are a few thoughts.

  • There is a bibtex package in R for importing bibtex files.
  • Various character functions could be used to extract the key words.
  • The data sounds a little like a two-mode network, which might mean packages like sna and igraph are useful.
  • Plots of 2d multidimensional scaling can also also be useful in visualising similarities (e.g., based on co-occurrence or some other measure) between words (here's a tutorial).
Improve this answer
$\endgroup$
5
$\begingroup$

so you have a document x keyword matrix which basically represents a bipartite graph (or two-mode network depending on your cultural background) with edges between documents and tags. If you're not interested in individual documents - as I understand you -, you can create a network of keywords by counting the number of cooccurrences between each keyword. Simply plotting this graph might already give you a neat idea of what this data looks like. You can further tweak the visualization if you, e.g., scale the size of the keywords by the number of total occurrences, or (in case you have a lot of keywords) introduce a minimum number of total occurrences for a keyword to appear in the first place.

As a tool, I can only recommend GraphViz which allows you to specify graphs like

keyword1 -- keyword2
keyword1 -- keyword3
keyword1[label="statistics", fontsize=...]

and "compile" them into pngs, pdfs, whatever, yielding very nice results (particularly if you play a bit with the font settings).

Improve this answer
$\endgroup$
1
$\begingroup$

I would recommend using Association Rule Learning for this. It allows you to find words that often co-occur.

If you have a lot of data, it will be much faster than calculating a correlation matrix.

See my video series on text mining here. Includes a tutorial on Association Rules for text.

Improve this answer
$\endgroup$
0
$\begingroup$

You may want to take a look at the phi coefficient which is a measure of association for nominal variables.

Improve this answer
$\endgroup$
0
$\begingroup$

You could try to employ the theory and praxis of association analysis or market basket analysis to your problem (just read "items" as "keywords" / "cited reference" and "market basket" as "journal article").

Disclaimer - this is just an idea, I did not do anything like that myself. Just my 2Cents.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.