Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

In R, I have a data frame comprising a class label C (a factor) and two measurements, M1 and M2. How do I compute the correlation between M1 and M2 within each class?

Ideally, I'd get back a data frame with one row for each class and two columns: the class label C and the correlation.


locked by whuber Oct 9 '15 at 13:38

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

up vote 20 down vote accepted

The package plyr is the way to go.

Here is a simple solution:

xx <- data.frame(group = rep(1:4, 100), a = rnorm(400) , b = rnorm(400) )

func <- function(xx)
return(data.frame(COR = cor(xx$a, xx$b)))

ddply(xx, .(group), func)

The output will be:

  group         COR
1     1  0.05152923
2     2 -0.15066838
3     3 -0.04717481
4     4  0.07899114
(+1) Nice plyr package, isn't it? :) – chl Oct 28 '10 at 8:47
This works great. Thanks for pointing out the plyr package! Could you please explain the ".(group)" syntax? – NPE Oct 28 '10 at 14:31
aix - sure. It means "split the data by the variable between .(), and on each subset perform the function". In order to have it include more variables, you should simply use this syntax: .(var1, var2, var3) . Which is like cutting your data by each combination of levels of var1, var2 and var3. And on each cut to perform your function. This package is maintained by Hadley (also th author of ggplot2), so I trust it will keep developing. – Tal Galili Oct 29 '10 at 6:54
Oh, and BTW, you could also use plyr with a parallel computing on several cores (almost automatically), see:… – Tal Galili Oct 29 '10 at 6:55
That's a nice answer, but I'm astonished there isn't a built-in solution for this, something like cor(x, y, by=z) would be so intuitive... – Waldir Leoncio Aug 1 '13 at 18:20

If you are inclined to use functions in the base package, you can use the by function, then reassemble the data:

xx <- data.frame(group = rep(1:4, 100), a = rnorm(400) , b = rnorm(400) )

# This returns a "by" object
result <- by(xx[,2:3], xx$group, function(x) {cor(x$a, x$b)})

# You get pretty close to what you want if you coerce it into a data frame via a matrix
result.dataframe <-

# Add the group column from the row names
result.dataframe$C <- rownames(result)
Nice, thanks! I've been experimenting with by, but couldn't figure out how to transform the result into a data frame. – NPE Oct 28 '10 at 14:38
+1, I certainly appreciate the use of a base function. – Patrick Coulombe Feb 14 '14 at 3:59

Another example using base packages and Tal's example data:

DataCov <- rbind, lapply( split(xx, xx$group),
             function(x) data.frame(group=x$group[1], mCov=cov(x$a, x$b)) ) )
Elegant solution Joshue. Do you think there are cases when one solution is better then another? – Tal Galili Oct 29 '10 at 16:05
I think it's a matter of preference. My example is essentially what plyr does but it gives you finer control, though it's not nearly as clean. My opinion would change if one solution had a better time/memory profile. I haven't compared them though. – Joshua Ulrich Oct 29 '10 at 16:08
How does this return the correlation? – what Aug 14 '15 at 10:44

Using data.table is shorter than dplyr

dt <- data.table(xx)
dtCor <- dt[, .(mCor = cor(M1,M2)), by=C]

Here's a similar method that will give you a table with the n's and p values for each correlation as well (rounded to 3 decimal places for convenience):

corrByGroup <- function(xx){
  return(data.frame(cbind(correl = round(rcorr(xx$a, xx$b)$r[1,2], digits=3),
                          n = rcorr(xx$a, xx$b)$n[1,2],
                          pvalue = round(rcorr(xx$a, xx$b)$P[1,2], digits=3))))

Here's a more modern solution, using the dplyr package (which didn't yet exist when the question was asked):

Construct the input:

xx <- data.frame(group = rep(1:4, 100), a = rnorm(400) , b = rnorm(400) )

Compute the correlations:

xx %>%
  group_by(group) %>%

The output:

Source: local data frame [4 x 2]

  group         COR
  (int)       (dbl)
1     1  0.05112400
2     2  0.14203033
3     3 -0.02334135
4     4  0.10626273

Not the answer you're looking for? Browse other questions tagged or ask your own question.