# Using correlate in Stata to compare within a variable

I am trying to compare correlations between subgroups of a variable. For instance, I am trying to compare the income of group 1 v. income of group 2 v. income of group 3 v. income of group 4, where the groups are specified by another categorical variable.

What I did was create new variables separating out the income variable into four new variables (i.e. income_1, income_2, income_3, income_4). Naturally, there are missing values that look something like this (each bracketed portion represents a new line where the '.' represents missing values):

[income_1   income_2    income_3   income_4]
[1 . . . ]
[1 . . .]
[. 2 . .]
[. 2 . .]
[. . 1 .]
[. . 1 .]
[. . . 2]
[. . . 2]


So when I try to perform correlate on these four values, I get an error that says 'no observations' probably because correlate performs listwise or pairwise deletion.

How can I fix this problem? I don't know how to get rid of the missing values.

• I am not quite sure I understand what kind of correlations you want to compute. It makes no sense to compute correlations by pairing the first observation of group 1 with the first observation in groups 2, 3 and 4, as the order of observations in cross-section data is usually arbitrary... unless you have a time series data, and these observations relate to the identical time periods. – StasK Jul 22 '13 at 19:29

As @Stask says in a comment, it doesn't really make sense to calculate a correlation in this case. Unless this is some kind of time-series/panel data, in which case the answer might be to do a reshape wide followed by a xpose or something else clever like that.

However, given that you want to do what you've described...

use http://www.ats.ucla.edu/stat/stata/notes/hsb2, clear


then:

gen write1=.
gen write2=.
gen write3 =.
replace write1=write if  write>30 & write <=40
replace write2=write if  write>40 & write <=50
replace write3=write if  write>50

putmata write1, omitmissing
getmata write1, replace force

putmata write2, omitmissing
getmata write2, replace force

putmata write3, omitmissing
getmata write3, replace force

list write1 write2 write3 in 1/10

+--------------------------+
| write1   write2   write3 |
|--------------------------|
1. |     33       44       52 |
2. |     39       46       59 |
3. |     40       46       52 |
4. |     37       49       52 |
5. |     38       49       59 |
|--------------------------|
6. |     31       44       57 |
7. |     31       44       55 |
8. |     31       41       65 |
9. |     40       47       60 |
10. |     33       41       63 |
+--------------------------+

cor  write1 write2 write3

|   write1   write2   write3
-------------+---------------------------
write1 |   1.0000
write2 |   0.2635   1.0000
write3 |   0.0393  -0.3456   1.0000


Not sure if that's what you want or not. Write1 only has 24 elements, while write3 has 126, and it appears that corr will thus only use the first 24 elements of each list in its calculations.

Obviously, you've also mangled write1, write2, and write3 so that they are in the wrong observations, so you'd want to drop write1 write2 write3 when you're done.

Perhaps you could do the corr strictly in Mata and avoid some messiness. I just got Stata a few days ago, so really don't know Mata yet.

Have you tried the by-functionality?

  sysuse auto, clear
bysort foreign : corr mpg weight

• I have, but my issue is that I want to compute the correlations among subgroups of one variable. So when I create new variables to separate the subgroups, I encounter problems with the missing values. – mwang2 Jul 22 '13 at 17:09
use http://www.ats.ucla.edu/stat/stata/notes/hsb2, clear


I will find the correlation of write for three different groups generated as follows:

gen write1=.
gen write2=.
gen write3 =.
replace write1=write if  write>30 & write <=40
replace write2=write if  write>40 & write <=50
replace write3=write if  write>50

list write1 write2 write3 in 1/10

list write1 write2 write3 in 1/10

+--------------------------+
| write1   write2   write3 |
|--------------------------|
1. |      .        .       52 |
2. |      .        .       59 |
3. |     33        .        . |
4. |      .       44        . |
5. |      .        .       52 |
|--------------------------|
6. |      .        .       52 |
7. |      .        .       59 |
8. |      .       46        . |
9. |      .        .       57 |
10. |      .        .       55 |
+--------------------------+

cor  write1 write2 write3
no observations
r(2000);


Yes, there are missing values so you can't compute correlation. Now instead of missing I assign 0 for all which doesn't satisfy the given if condition.

gen write1=0
gen write2=0
gen write3 =0
replace write1=write if  write>30 & write <=40
replace write2=write if  write>40 & write <=50
replace write3=write if  write>50
list write1 write2 write3 in 1/10

+--------------------------+
| write1   write2   write3 |
|--------------------------|
1. |     31        0        0 |
2. |     36        0        0 |
3. |     39        0        0 |
4. |     39        0        0 |
5. |     38        0        0 |
|--------------------------|
6. |     33        0        0 |
7. |     35        0        0 |
8. |     37        0        0 |
9. |     33        0        0 |
10. |     31        0        0 |

cor  write1 write2 write3
(obs=200)

|   write1   write2   write3
-------------+---------------------------
write1 |   1.0000
write2 |  -0.2117   1.0000
write3 |  -0.4760  -0.7454   1.0000

• This is not going to work. Suppose I increased all the values of write variable by 1000. This would preserve the underlying correlations within groups, as they are not affected by the mean change. However, in your answer, all correlations will become close to -1, as they are based on deviation of 0 and 1000-something from the mean of your constructed variable padded with zeroes; and a 1000-something value of one of the group-wise variables is always associated with a zero value on all other variables. In fact, you do see some of that in your output already. – StasK Jul 22 '13 at 16:45
• You are right! I am just showing the OP that the problem is due to missing observations, but I haven't vouched that the later method is good to follow since you can't assign 0 to all missings (and infact these are not the missings in the actual data) – Metrics Jul 22 '13 at 16:50
• Thanks for showing that my problem is due to the missing observations! Do you know of a better way to go about this issue I am having then? I realize that I can't just assign zero to all the missing values because some of the actual values could be zero. – mwang2 Jul 22 '13 at 17:14
• First, correlation requires the number of observations (n) to be the same for each subgroup. If that is not the case, then it will take minimum n (in Stata) and do correlation which is not good. In the example above, n1=24, n2=50, and n3=126. So, correlation is obtained using only 24 observations which you may not want. – Metrics Jul 22 '13 at 17:37