Tool for generating correlated data sets

Question

Does anyone know of a tool that I can use to generate a set of data with known correlations (and to put the icing on the cake - output this in json,csv,txt or some common format)?

I am working on some data visualizations and want to evaluate which ones can more easily allow a user to spot correlations - visually.

Answer 1

You could do it in any variety of places. Excel, R, ... almost anything capable of doing basic statistical calculations.

1. Population correlation. This is a simple matter in the bivariate case of taking independent random variables with the same standard deviation and creating a third variable from those two that has the required correlation with one of the two random variables. If $X_1$ and $X_2$ are independent standard normal variables, then $Y=rX_2+\sqrt{1-r^2}X_1$ will have correlation $r$ between $Y$ and $X_2$.

   Here's an example in R:

    n = 10
    r = 0.8
    x1 = rnorm(n)
    x2 = rnorm(n)
    y1 = r*x2+sqrt(1-r*r)*x1   
   

   Here the underlying variables have population correlation of the desired size, but the sample correlation will differ from it. (I just ran the code three times and got sample correlations of 0.938,0.895, and 0.933).

   This could be done in Excel or any number of other packages with similar ease.

   If you need it for more than two variables and some prespecified correlation matrix, this can be done using Cholesky decomposition (of which the above is a special case). If $Z$ is a vector of length $k$ of independent random variables with unit (or at least constant) standard deviation; and $\S$ is a correlation matrix with Cholesky decomposition $S=LL'$, then $LZ$ with have population correlation $S$.

2. Sample correlation. For the exact sample correlation, you need samples with exactly zero sample correlation, and identical sample variances, before applying the above trick. There are a variety of ways to achieve that, but one simple way is to take residuals from a regression (which will be uncorrelated with the x-variable in the regression), and then scale both variables to have unit variance.

   Here's an example in R:

    n = 10
    r = 0.8
    x1 = rnorm(n)
    x2 = rnorm(n)
    y1 = scale(x2) * r  +  scale(residuals(lm(x1~x2))) * sqrt(1-r*r)
   

   which produces the correlation:

    cor(y1,x2)
        [,1]
   [1,]  0.8
   

   exactly as desired.

---

So now it's just a matter of writing out the results in your preferred format (all the formats you mention can be done easily; for example, as a csv file, you'd call write.csv:

write.csv(data.frame(y=y1,x=x2),file="myfile.csv")

which makes a file of the name "myfile.csv" in the current working directory with the contents:

"","y","x"
"1",0.743433299251026,0.617686871809365
"2",0.527604385327034,-0.113047553664104
"3",-0.397333571358269,0.196447643803443
"4",-0.875264248799599,-1.57628371273354
"5",-0.225441433921137,-0.107919886825751
"6",0.0817573026498336,0.370207951209058
"7",-2.15935431462587,-1.21145928947767
"8",1.46638207013879,1.10215217029937
"9",0.311683673588212,-0.470550477344661
"10",0.526532837749974,-0.104382608454622

Answer 2

Package mvtnorm in R produces random multivariate normals. You can specify the correlations.

If M is your matrix of random normals, do write.csv(M, file="mydata.csv") to write it out to a file.

Answer 3

Just to prevent to set correlation which are "impossible" as a whole set (the matrix of correlations can become non-positivedefinite) - for instance you can't define two nearly correlated variables and a third one near to one of them and far to the other of them - it might be more useful to begin with a "factorloadings"-matrix instead, which describe the composition of the randomvariables as linear (regression)-equations. This is less "natural" to look at at the beginning but one can get used to this.
The following might be done similarly, and perhaps better, in R but I show it here in my own matrix-tool-language MatMate because I'm unexperienced in R. It could be done shorter, without the naming/the use of variables like N , nv, etc, you could just insert the values but for documentation here I've done it with that richer documented form. Example is :

• 3 hidden common factors and
• 6 itemspecific error-factors (normal distribution) make
• 6 "empirical" variables

• measured in N=1000 cases.

  //==============================================================   
  N = 1000       
  nv = 6          // set number of empirical variables               
  ncf,nef = 3,nv  // set number of common factors, error-factors               
  nf = ncf+nef    // needed uncorrelated random-factors                  
  
  // create a hidden ("unknown") loadingsmatrix, which describes the 
  // composition of our empirical data by the "unknown" factors
  // remember we want ncf=3 common factors and nef=nv=6 error factors
  ulad = {{ 10.0 ,  1,  0}, _
          {  9   ,  0,  1}, _
          {  0   , 11,  0}, _
          {  1   , 12,  1}, _
          {  0.2 , -1, 11}, _
          { -0.3 ,  1, 10}}
  
  ulad = ulad || 2*einh(nef)  // append a identity-matrix as definition of the 
                              // error-variance
                              // make the itemspecific variance a bit bigger
                              // than the spurious cross-factors loadings in
                              // the ulad-loadingsmatrix 
       chk = ulad * ulad'     // check the expected covariancematrix
       list chk               // print it out
       chk = covtocorr (chk)  // look at it as correlation-matrix
       list chk               // print it out
  
  
  // Now generate random data for nf uncorrelated normally-distributed factors
   set randomstart=41  // set randomgenerator to get reproducable random data
   rn = randomn(nf,N)  // fix a basic datamatrix of random numbers (normal dist)
      chk = (rn *' - N*einh(nf))*1e3  // we find spurious correlations of 1e-3
  
   ufac=unkorrzl(rn)        // refine data in rn: remove spurious correlations
      // the process leaves still spurious correlations of 1e-12
      chk = (ufac *' - N*einh(nf))*1e12  // still spurious correlations of 1e-12
  
      // repeat to higher-precision 
            ufac=zvaluezl(abwzl(ufac))  // correct again for exacter z-values
      ufac=unkorrzl(ufac)   // remove again spurious correlation
      chk = (ufac *' - N*einh(nf))*1e18  // spurious correlations of 1e-18
  
  // create "empirical" dataset with N=1000 measures
  //       having the wished compositions of the random factors 
  data = ulad * ufac
  
  // ========= end of the empirically unobservable mechanism ============
  
  // now you can proceed with regression, factoranalysis or whatever on
  // that data      
  // .....................
  // or you can write out the data in a csv-file or into the clipboard
  matwrite csv("mydata.csv",10,6) = data   // write in csv-format, cases along row
                                           // max 10 digits, 6 of them decimals
  matwrite csv("mydata.csv",10,6) = data'  // cases along column
  matwrite csv("clip",10,6) = data'  // write it directly into clipboard
                                     // to insert it, for instance, in Excel