5
$\begingroup$

Does anyone know of a tool that I can use to generate a set of data with known correlations (and to put the icing on the cake - output this in json,csv,txt or some common format)?

I am working on some data visualizations and want to evaluate which ones can more easily allow a user to spot correlations - visually.

$\endgroup$
  • 1
    $\begingroup$ Do you want a specified population correlation (which the sample correlation would be close to) or a specified sample correlation? $\endgroup$ – Glen_b Aug 15 '14 at 3:48
  • $\begingroup$ A tool that can generate either would be fine although the latter would be better. $\endgroup$ – alaknis Aug 16 '14 at 12:07
  • 1
    $\begingroup$ It turns out that answers to this question just about cover what you need (though it's specifically an R question, there's a fair bit of detail in some of the answers). It's not really a duplicate, but might be near enough for this to close - I am unsure, but hold off for now on the basis that sometimes it's useful to have different versions of somewhat similar questions. $\endgroup$ – Glen_b Aug 17 '14 at 3:11
7
$\begingroup$

You could do it in any variety of places. Excel, R, ... almost anything capable of doing basic statistical calculations.

  1. Population correlation. This is a simple matter in the bivariate case of taking independent random variables with the same standard deviation and creating a third variable from those two that has the required correlation with one of the two random variables. If $X_1$ and $X_2$ are independent standard normal variables, then $Y=rX_2+\sqrt{1-r^2}X_1$ will have correlation $r$ between $Y$ and $X_2$.

    Here's an example in R:

     n = 10
     r = 0.8
     x1 = rnorm(n)
     x2 = rnorm(n)
     y1 = r*x2+sqrt(1-r*r)*x1   
    

    Here the underlying variables have population correlation of the desired size, but the sample correlation will differ from it. (I just ran the code three times and got sample correlations of 0.938,0.895, and 0.933).

    This could be done in Excel or any number of other packages with similar ease.

    If you need it for more than two variables and some prespecified correlation matrix, this can be done using Cholesky decomposition (of which the above is a special case). If $Z$ is a vector of length $k$ of independent random variables with unit (or at least constant) standard deviation; and $\S$ is a correlation matrix with Cholesky decomposition $S=LL'$, then $LZ$ with have population correlation $S$.

  2. Sample correlation. For the exact sample correlation, you need samples with exactly zero sample correlation, and identical sample variances, before applying the above trick. There are a variety of ways to achieve that, but one simple way is to take residuals from a regression (which will be uncorrelated with the x-variable in the regression), and then scale both variables to have unit variance.

    Here's an example in R:

     n = 10
     r = 0.8
     x1 = rnorm(n)
     x2 = rnorm(n)
     y1 = scale(x2) * r  +  scale(residuals(lm(x1~x2))) * sqrt(1-r*r)
    

    which produces the correlation:

     cor(y1,x2)
         [,1]
    [1,]  0.8
    

    exactly as desired.


So now it's just a matter of writing out the results in your preferred format (all the formats you mention can be done easily; for example, as a csv file, you'd call write.csv:

write.csv(data.frame(y=y1,x=x2),file="myfile.csv")

which makes a file of the name "myfile.csv" in the current working directory with the contents:

"","y","x"
"1",0.743433299251026,0.617686871809365
"2",0.527604385327034,-0.113047553664104
"3",-0.397333571358269,0.196447643803443
"4",-0.875264248799599,-1.57628371273354
"5",-0.225441433921137,-0.107919886825751
"6",0.0817573026498336,0.370207951209058
"7",-2.15935431462587,-1.21145928947767
"8",1.46638207013879,1.10215217029937
"9",0.311683673588212,-0.470550477344661
"10",0.526532837749974,-0.104382608454622
$\endgroup$
3
$\begingroup$

Package mvtnorm in R produces random multivariate normals. You can specify the correlations.

If M is your matrix of random normals, do write.csv(M, file="mydata.csv") to write it out to a file.

$\endgroup$
  • $\begingroup$ Not familiar with R but trying this out $\endgroup$ – alaknis Aug 16 '14 at 12:08
1
$\begingroup$

Just to prevent to set correlation which are "impossible" as a whole set (the matrix of correlations can become non-positivedefinite) - for instance you can't define two nearly correlated variables and a third one near to one of them and far to the other of them - it might be more useful to begin with a "factorloadings"-matrix instead, which describe the composition of the randomvariables as linear (regression)-equations. This is less "natural" to look at at the beginning but one can get used to this.
The following might be done similarly, and perhaps better, in R but I show it here in my own matrix-tool-language MatMate because I'm unexperienced in R. It could be done shorter, without the naming/the use of variables like N , nv, etc, you could just insert the values but for documentation here I've done it with that richer documented form. Example is :

  • 3 hidden common factors and
  • 6 itemspecific error-factors (normal distribution) make
  • 6 "empirical" variables
  • measured in N=1000 cases.

    //==============================================================   
    N = 1000       
    nv = 6          // set number of empirical variables               
    ncf,nef = 3,nv  // set number of common factors, error-factors               
    nf = ncf+nef    // needed uncorrelated random-factors                  
    
    // create a hidden ("unknown") loadingsmatrix, which describes the 
    // composition of our empirical data by the "unknown" factors
    // remember we want ncf=3 common factors and nef=nv=6 error factors
    ulad = {{ 10.0 ,  1,  0}, _
            {  9   ,  0,  1}, _
            {  0   , 11,  0}, _
            {  1   , 12,  1}, _
            {  0.2 , -1, 11}, _
            { -0.3 ,  1, 10}}
    
    ulad = ulad || 2*einh(nef)  // append a identity-matrix as definition of the 
                                // error-variance
                                // make the itemspecific variance a bit bigger
                                // than the spurious cross-factors loadings in
                                // the ulad-loadingsmatrix 
         chk = ulad * ulad'     // check the expected covariancematrix
         list chk               // print it out
         chk = covtocorr (chk)  // look at it as correlation-matrix
         list chk               // print it out
    
    
    // Now generate random data for nf uncorrelated normally-distributed factors
     set randomstart=41  // set randomgenerator to get reproducable random data
     rn = randomn(nf,N)  // fix a basic datamatrix of random numbers (normal dist)
        chk = (rn *' - N*einh(nf))*1e3  // we find spurious correlations of 1e-3
    
     ufac=unkorrzl(rn)        // refine data in rn: remove spurious correlations
        // the process leaves still spurious correlations of 1e-12
        chk = (ufac *' - N*einh(nf))*1e12  // still spurious correlations of 1e-12
    
        // repeat to higher-precision 
              ufac=zvaluezl(abwzl(ufac))  // correct again for exacter z-values
        ufac=unkorrzl(ufac)   // remove again spurious correlation
        chk = (ufac *' - N*einh(nf))*1e18  // spurious correlations of 1e-18
    
    // create "empirical" dataset with N=1000 measures
    //       having the wished compositions of the random factors 
    data = ulad * ufac
    
    // ========= end of the empirically unobservable mechanism ============
    
    // now you can proceed with regression, factoranalysis or whatever on
    // that data      
    // .....................
    // or you can write out the data in a csv-file or into the clipboard
    matwrite csv("mydata.csv",10,6) = data   // write in csv-format, cases along row
                                             // max 10 digits, 6 of them decimals
    matwrite csv("mydata.csv",10,6) = data'  // cases along column
    matwrite csv("clip",10,6) = data'  // write it directly into clipboard
                                       // to insert it, for instance, in Excel
    
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.