7
$\begingroup$

I am building a big data frame by merging the content of a few files together. These files share the same columns layout.

c = read.delim('bigfile1.txt')
c1 = read.delim('bigfile2.txt')
c2 = read.delim('bigfile3.txt')

ctmp1 = merge(c, c1, all=TRUE)
ctmp2 = merge(ctmp1, c2, all=TRUE)

Is the above code efficient?

Should I reuse the same variable name instead, e.g.

tmp = merge(c, c1, all=TRUE)
tmp = merge(tmp, c2, all=TRUE)
$\endgroup$
9
  • 2
    $\begingroup$ What kind of efficiency are you looking for? Speed or memory usage? $\endgroup$
    – mpiktas
    Aug 16 '11 at 7:54
  • $\begingroup$ mptiktas: Good question! Performance first, memory footprint second. $\endgroup$ Aug 16 '11 at 9:17
  • 2
    $\begingroup$ I doubt it would have a serious effect on performance, but I would read files into a list then use rbind or cbind through do.call to "merge". That way you don't have extra objects lying around. $\endgroup$ Aug 16 '11 at 9:32
  • $\begingroup$ If things are as I suspect (see my answer), you may even be better off joining the files outside of R and then reading them in: performance will be good, and you can probably avoid the need to have more than 1 file (the one you're currently adding to the resulting file) in memory at the same time. $\endgroup$
    – Nick Sabbe
    Aug 16 '11 at 10:05
  • $\begingroup$ @Nick: totally agree with your suggestion. In this particular case I only have read access to the source folder. Of course, I could cp the files to a local location and cat them together. $\endgroup$ Aug 16 '11 at 13:00
12
$\begingroup$

You could cat them within R as follows:

read.table(pipe("cat bigfile1.txt bigfile2.txt bigfile3.txt"))
$\endgroup$
9
$\begingroup$

Since you mention that they have the same column layout, you probably want the three (or more) data.frames to be appended below eachother, right?

In that case, you can look at rbind:

cres = rbind(c, c1, c2)

Beware, though: with a lot of data.frames, I've noticed the performance to be subpar (this has to do with the way data.frames are managed in-memory, as lists of columns). Also, there may be issues with factors: having the same column layout, but holding different levels for factors may break this (haven't tried).

$\endgroup$
1
  • 2
    $\begingroup$ Note: if your data is all numerical, convert each data.frame to matrix first, then do an rbind, and turn the result into a data.frame again. $\endgroup$
    – Nick Sabbe
    Aug 16 '11 at 9:35
3
$\begingroup$

Check out rbind.fill from plyr package. I've recently seen Hadley's comment that it is efficient but unable to find it.

$\endgroup$
2
$\begingroup$

If by efficient you mean "fast," check out the data.table package. It has very fast mergers.

$\endgroup$
2
$\begingroup$

You can try join statement in sqldf package. I find working with SQL in case large dataset much easier. Please find the link here for reference

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.