# Tool for generating correlated data sets

Does anyone know of a tool that I can use to generate a set of data with known correlations (and to put the icing on the cake - output this in json,csv,txt or some common format)?

I am working on some data visualizations and want to evaluate which ones can more easily allow a user to spot correlations - visually.

• Do you want a specified population correlation (which the sample correlation would be close to) or a specified sample correlation? – Glen_b Aug 15 '14 at 3:48
• A tool that can generate either would be fine although the latter would be better. – alaknis Aug 16 '14 at 12:07
• It turns out that answers to this question just about cover what you need (though it's specifically an R question, there's a fair bit of detail in some of the answers). It's not really a duplicate, but might be near enough for this to close - I am unsure, but hold off for now on the basis that sometimes it's useful to have different versions of somewhat similar questions. – Glen_b Aug 17 '14 at 3:11

You could do it in any variety of places. Excel, R, ... almost anything capable of doing basic statistical calculations.

1. Population correlation. This is a simple matter in the bivariate case of taking independent random variables with the same standard deviation and creating a third variable from those two that has the required correlation with one of the two random variables. If $X_1$ and $X_2$ are independent standard normal variables, then $Y=rX_2+\sqrt{1-r^2}X_1$ will have correlation $r$ between $Y$ and $X_2$.

Here's an example in R:

 n = 10
r = 0.8
x1 = rnorm(n)
x2 = rnorm(n)
y1 = r*x2+sqrt(1-r*r)*x1


Here the underlying variables have population correlation of the desired size, but the sample correlation will differ from it. (I just ran the code three times and got sample correlations of 0.938,0.895, and 0.933).

This could be done in Excel or any number of other packages with similar ease.

If you need it for more than two variables and some prespecified correlation matrix, this can be done using Cholesky decomposition (of which the above is a special case). If $Z$ is a vector of length $k$ of independent random variables with unit (or at least constant) standard deviation; and $\S$ is a correlation matrix with Cholesky decomposition $S=LL'$, then $LZ$ with have population correlation $S$.

2. Sample correlation. For the exact sample correlation, you need samples with exactly zero sample correlation, and identical sample variances, before applying the above trick. There are a variety of ways to achieve that, but one simple way is to take residuals from a regression (which will be uncorrelated with the x-variable in the regression), and then scale both variables to have unit variance.

Here's an example in R:

 n = 10
r = 0.8
x1 = rnorm(n)
x2 = rnorm(n)
y1 = scale(x2) * r  +  scale(residuals(lm(x1~x2))) * sqrt(1-r*r)


which produces the correlation:

 cor(y1,x2)
[,1]
[1,]  0.8


exactly as desired.

So now it's just a matter of writing out the results in your preferred format (all the formats you mention can be done easily; for example, as a csv file, you'd call write.csv:

write.csv(data.frame(y=y1,x=x2),file="myfile.csv")


which makes a file of the name "myfile.csv" in the current working directory with the contents:

"","y","x"
"1",0.743433299251026,0.617686871809365
"2",0.527604385327034,-0.113047553664104
"3",-0.397333571358269,0.196447643803443
"4",-0.875264248799599,-1.57628371273354
"5",-0.225441433921137,-0.107919886825751
"6",0.0817573026498336,0.370207951209058
"7",-2.15935431462587,-1.21145928947767
"8",1.46638207013879,1.10215217029937
"9",0.311683673588212,-0.470550477344661
"10",0.526532837749974,-0.104382608454622


Package mvtnorm in R produces random multivariate normals. You can specify the correlations.

If M is your matrix of random normals, do write.csv(M, file="mydata.csv") to write it out to a file.

• Not familiar with R but trying this out – alaknis Aug 16 '14 at 12:08

Just to prevent to set correlation which are "impossible" as a whole set (the matrix of correlations can become non-positivedefinite) - for instance you can't define two nearly correlated variables and a third one near to one of them and far to the other of them - it might be more useful to begin with a "factorloadings"-matrix instead, which describe the composition of the randomvariables as linear (regression)-equations. This is less "natural" to look at at the beginning but one can get used to this.
The following might be done similarly, and perhaps better, in R but I show it here in my own matrix-tool-language MatMate because I'm unexperienced in R. It could be done shorter, without the naming/the use of variables like N , nv, etc, you could just insert the values but for documentation here I've done it with that richer documented form. Example is :

• 3 hidden common factors and
• 6 itemspecific error-factors (normal distribution) make
• 6 "empirical" variables
• measured in N=1000 cases.

//==============================================================
N = 1000
nv = 6          // set number of empirical variables
ncf,nef = 3,nv  // set number of common factors, error-factors
nf = ncf+nef    // needed uncorrelated random-factors

// composition of our empirical data by the "unknown" factors
// remember we want ncf=3 common factors and nef=nv=6 error factors
ulad = {{ 10.0 ,  1,  0}, _
{  9   ,  0,  1}, _
{  0   , 11,  0}, _
{  1   , 12,  1}, _
{  0.2 , -1, 11}, _
{ -0.3 ,  1, 10}}

ulad = ulad || 2*einh(nef)  // append a identity-matrix as definition of the
// error-variance
// make the itemspecific variance a bit bigger
list chk               // print it out
chk = covtocorr (chk)  // look at it as correlation-matrix
list chk               // print it out

// Now generate random data for nf uncorrelated normally-distributed factors
set randomstart=41  // set randomgenerator to get reproducable random data
rn = randomn(nf,N)  // fix a basic datamatrix of random numbers (normal dist)
chk = (rn *' - N*einh(nf))*1e3  // we find spurious correlations of 1e-3

ufac=unkorrzl(rn)        // refine data in rn: remove spurious correlations
// the process leaves still spurious correlations of 1e-12
chk = (ufac *' - N*einh(nf))*1e12  // still spurious correlations of 1e-12

// repeat to higher-precision
ufac=zvaluezl(abwzl(ufac))  // correct again for exacter z-values
ufac=unkorrzl(ufac)   // remove again spurious correlation
chk = (ufac *' - N*einh(nf))*1e18  // spurious correlations of 1e-18

// create "empirical" dataset with N=1000 measures
//       having the wished compositions of the random factors