# Software needed to scrape data from graph

Anybody have any experience with software (preferably free, preferably open source) that will take an image of data plotted on cartesian coordinates (a standard, everyday plot) and extract the coordinates of the points plotted on the graph?

Essentially, this is a data-mining problem and a reverse data-visualization problem.

-
For one solution, see the comments to this reply. Open source solutions would include image processing or raster GIS software (GRASS is a likely candidate) or, perhaps, GNU Octave. I'm mentioning these as a comment because I haven't used either for this specific purpose, so please take them as possibilities, not as definite solutions. –  whuber Aug 18 '11 at 4:20
I'm hoping for code/software specifically for scraping graphs, and I remember such packages existed, at least they did 10 yrs ago, but I can't remember their names now, and don't know if they work on current operating systems. –  Alex Holcombe Aug 18 '11 at 4:56
@Alex, try googling "Graph Digitizer Open Source" –  David Aug 18 '11 at 5:52
It would be great if the answers also explained how it works or how well. Now I just have to investigate all the links to judge whether I should upvote or not –  Ivo Flipse Aug 18 '11 at 12:58
@Ivo I added an explanation to my answer –  David Aug 19 '11 at 3:13

Check out the digitize package for R. Its designed to solve exactly this sort of problem.

/edit: No longer on CRAN, but you can still get it from R-Forge.

-
That worked, and is perfect because I use R! –  Alex Holcombe Aug 18 '11 at 10:40
@Alex Holcombe glad to help out! I stumbled across digitize a while ago, and thought it was pretty cool. I haven't needed to use it yet, but it's on my mental list of "cool R packages." –  Zach Aug 18 '11 at 18:06
There is a nice article / tutorial in R Journal, June 2011 –  David Aug 23 '11 at 22:56
The digitize package is no longer available... :-( –  Adam Ryczkowski Dec 1 '12 at 7:09
One can still get it but it has two problems. It depends on another archived package ReadImages which cannot be loaded in R 3.0.2 , because (I suspect) it doesn't have a NAMESPACE. Likewise digitize fails to load under R 3.0.2. –  DWin Oct 19 '13 at 19:02
show 1 more comment

What you are looking for is graph digitizing software. They all work pretty much the same:

1. upload an image
2. set the x and y scales by indicating the values at two points on each axis
3. indicate if the scale is linear, log, etc,
4. click on the points.
• Some of the programs automatically recognize lines or points. I am usually after points, and I find them too inconsistent to be helpful even with 100s of points. I have not found one that recognizes different symbols. This feature could be worth the trouble for digitizing lines, but I have never had to do this.

The program returns each point as an x-y matrix.

Often it helps selecting points if the image is zoomed, either by uploading a zoomed version of the image or using the zooming feature available in some of the programs.

I have experience using the following programs:

• Digitizer (shareware) auto point / line recognition. Available in Ubuntu repository (engauge-digitizer)
• Get Data (shareware) has zoom window, auto point / line recognition
• DigitizeIt (shareware) auto point / line recognition
• ImageJ (open source, most extensible after R digitize)
• R digitize (free, open source), because it simplifies the processs of getting data from the graph into an analysis by keeping all of the steps in R. See the tutorial in R-Journal
• GrabIt! (free demo, $69) Excel plug-in Other options I have not used: • WebPlotDigitzer (free, online). Extracts data from images. Demo here. • GraphClick (Mac,$8)
• g3data (open source - GNU GPL) Has zoom window, no auto-recognition. Available in Ubuntu repository.

All of these worked fine. Except in contexts where measurement error is very small, error from graph scraping is insignificant (e.g. error from digitization << size of error bars or uncertainty in the estimate). If have not tested the accuracy of any of these programs, but it would be interesting to compare among users, among programs, and against the results of reproduced statistical analyses.

-
Awesome edit @David, that was a great improvement! –  Ivo Flipse Oct 12 '11 at 19:26
+1 for the effort put into explanation. –  naught101 Apr 20 '12 at 1:40
g3data (open source - GNU GPL) has zoom window, no auto-recognition. Available in Ubuntu repository. I can't compare, as it's the only one I've tried; but I found it very easy to use. –  Scortchi Oct 16 '13 at 22:54

Other answerers assume that you deal with raster image of a graph. But nowadays the good practice is to publish graphs in vector form. In this case you can achieve much higher exactness of the recovered data and even estimate the recovery error if you work with the code of the vector graph directly, without converting it to raster image.

Since the papers are published online as PDF files, I assume that you have a PDF file which contains vector plot with data you wish to recover from it (get in numerical form) and estimate introduced recovery error.

First of all, PDF is a vector format which is basically textual (can be read by a text editor). The problem is that it can (and almost always) contain compressed data streams which require to be uncompressed in order to read them by a text editor. These compressed data streams usually contain the information we need.

There are several ways to uncompress data streams in order to convert PDF file to a textual document with readable PDF code. Probably the simplest way is to use free QPDF utility with --stream-data=uncompress option:

qpdf infile.pdf --stream-data=uncompress -- outfile.pdf


Some other ways are described here and here.

The generated outfile.pdf can be opened by a text editor. Now you need PDF Reference Manual to understand what you see. Do not panic at this moment! In really you need to know only few operators described in the section "8.6.1 Path segment operators" on the pages 224 - 228. The most important operators are (the first column contains coordinate specification for an operator, the second contains the operator and the third is operator name):

x y               m   moveto

x y               l   lineto

x y width height  re  rectangle

h   closepath


In most cases it is sufficient to know these four operators for recovering the data.

Now you need to import the outfile.pdf file as text into some program where you can manipulate the data. I'll show how to do it with Mathematica.

Importing the file:

pdfCode = Import["outfile.pdf", "Text"];


Now I assume the simplest case: the graph contains a line which consists of many two-point segments. In this case each segment of the line is encoded like this:

268.79999 408.92975 m
272.39999 408.92975 l


Extracting all such segments from the PDF code:

lines = StringCases[pdfCode,
StartOfLine ~~ x1 : NumberString ~~ " " ~~ y1 : NumberString ~~ " m\n" ~~
x2 : NumberString ~~ " " ~~ y2 : NumberString ~~ " l\n"
:> ToExpression@{{x1, y1}, {x2, y2}}];


Visualizing them:

Graphics[{Line[lines]}]


You get something like this (the paper I am working with contains four graphs):

Each two adjacent segments share one point. So in this case you can turn the sequences of adjacent segments into paths:

paths = Split[lines, #1[[2]] == #2[[1]] &];


Now you can visualize all the paths separately:

Graphics[{Line /@ paths}]


From this figure you can select (by double-clicking) the path you are looking for, copy graphics selection and paste as new Graphics. For converting it backward to list of points you take the element {1, 1, 1}. Now we have the points but not in the coordinate system of the graph but in the coordinate system of the PDF file. We need to establish relationship between them.

From the above plot you select ticks by hand (holding Shift for multiple selection), then copy them and paste as new Graphics. Here is how you can extract coordinates of horizontal ticks:

Now check the differences between ticks:

Differences[reHorTicks]


From these differences you can see how precise is positioning of the ticks in the PDF file. It gives an estimate of error introduced by converting original datapoints into vector graph included in the PDF file. If there are appreciable errors in ticks positioning you can reduce the error by fitting the coordinates of ticks to a linear model. This linear function now can be used to get original coordinates of points of the path (that is in the coordinate system of the plot).

-
+1 Very creative and potentially quite helpful; thank you. –  whuber Jan 14 at 19:36