Tool for generating correlated data sets Does anyone know of a tool that I can use to generate a set of data with known correlations (and to put the icing on the cake - output this in json,csv,txt or some common format)?
I am working on some data visualizations and want to evaluate which ones can more easily allow a user to spot correlations - visually.
 A: Package mvtnorm in R produces random multivariate normals. You can specify the correlations.
If M is your matrix of random normals, do write.csv(M, file="mydata.csv") to write it out to a file.
A: Just to prevent to set correlation which are "impossible" as a whole set (the matrix of correlations can become non-positivedefinite) - for instance you can't define two nearly correlated variables and a third one near to one of them and far to the other of them - it might be more useful to begin with a "factorloadings"-matrix instead, which describe the composition of the randomvariables as linear (regression)-equations. This is less "natural" to look at at the beginning but one can get used to this.
The following might be done similarly, and perhaps better, in R but I show it here in my own matrix-tool-language MatMate because I'm unexperienced in R. It could be done shorter, without the naming/the use of variables like N , nv, etc, you could just insert the values but for documentation here I've done it with that richer documented form. Example is :      


*

*3 hidden common factors and         

*6 itemspecific error-factors (normal distribution)  make         

*6 "empirical" variables             

*measured in N=1000 cases.                
//==============================================================   
N = 1000       
nv = 6          // set number of empirical variables               
ncf,nef = 3,nv  // set number of common factors, error-factors               
nf = ncf+nef    // needed uncorrelated random-factors                  

// create a hidden ("unknown") loadingsmatrix, which describes the 
// composition of our empirical data by the "unknown" factors
// remember we want ncf=3 common factors and nef=nv=6 error factors
ulad = {{ 10.0 ,  1,  0}, _
        {  9   ,  0,  1}, _
        {  0   , 11,  0}, _
        {  1   , 12,  1}, _
        {  0.2 , -1, 11}, _
        { -0.3 ,  1, 10}}

ulad = ulad || 2*einh(nef)  // append a identity-matrix as definition of the 
                            // error-variance
                            // make the itemspecific variance a bit bigger
                            // than the spurious cross-factors loadings in
                            // the ulad-loadingsmatrix 
     chk = ulad * ulad'     // check the expected covariancematrix
     list chk               // print it out
     chk = covtocorr (chk)  // look at it as correlation-matrix
     list chk               // print it out


// Now generate random data for nf uncorrelated normally-distributed factors
 set randomstart=41  // set randomgenerator to get reproducable random data
 rn = randomn(nf,N)  // fix a basic datamatrix of random numbers (normal dist)
    chk = (rn *' - N*einh(nf))*1e3  // we find spurious correlations of 1e-3

 ufac=unkorrzl(rn)        // refine data in rn: remove spurious correlations
    // the process leaves still spurious correlations of 1e-12
    chk = (ufac *' - N*einh(nf))*1e12  // still spurious correlations of 1e-12

    // repeat to higher-precision 
          ufac=zvaluezl(abwzl(ufac))  // correct again for exacter z-values
    ufac=unkorrzl(ufac)   // remove again spurious correlation
    chk = (ufac *' - N*einh(nf))*1e18  // spurious correlations of 1e-18

// create "empirical" dataset with N=1000 measures
//       having the wished compositions of the random factors 
data = ulad * ufac

// ========= end of the empirically unobservable mechanism ============

// now you can proceed with regression, factoranalysis or whatever on
// that data      
// .....................
// or you can write out the data in a csv-file or into the clipboard
matwrite csv("mydata.csv",10,6) = data   // write in csv-format, cases along row
                                         // max 10 digits, 6 of them decimals
matwrite csv("mydata.csv",10,6) = data'  // cases along column
matwrite csv("clip",10,6) = data'  // write it directly into clipboard
                                   // to insert it, for instance, in Excel

