# Method to test whether a small number of perfect correlations can be trusted within a large sample

There are two binary vectors with 0s or 1s as values and the correlation between them is calculated; this is done for 20,000 pairs of vectors. Theoretically, is there a difference between having only 3 of these 20,000 correlations having perfect correlation vs. 15,000 of these 20,000 having perfect correlation? Can the perfect correlations in either of these scenarios be trusted more?

Is there a numerical method to actually test whether or not these perfect correlations are meaningful or trustworthy? For instance, after a certain sample size, could having only a couple perfect correlations out of thousands or millions of correlations be caused solely by chance?

Edit for more context:

I'm interested in comparing two binary vectors for utility meters (for instance, 0 = the meter is on, 1 = the meter is off). The correlations between these vectors are being used to see if these meters are most likely on the same transformers or not. So if we have 20,000 correlations between different meters and only 3 of them are perfect or high (0.9-1.0) correlations, can that be trusted versus maybe 15,000 of them being perfect or high correlations? And is there a numerical method in order to test if these perfect correlations are legitimate?

• Your intuition is good, but I think the question as it stands is too abstract yet non-technical for me to give a good answer. Can you edit the question to provide more context? What set you to think about this, and what larger problem are you trying to solve? May 18, 2016 at 20:15
• Could you explain what you mean by a correlation to be "legitimate" or "trusted"? Do you have data quality concerns, perhaps? Your underlying problem is of interest because it is the same one shared, for example, by standardized testing companies. A multiple-choice standardized test can be considered a binary vector (1 when an answer is marked, 0 for all other possible answers). Strong "correlations" can be taken as evidence of copying. The literature on testing might give you some useful ideas about good ways to assess all these pairs and how to decide which ones are from common transformers.
– whuber
May 18, 2016 at 22:03
• Some meters are incorrectly labeled as being on the same transformer when they aren't or vice versa, so the correlations are being done in order to find out which of these meters are being incorrectly labeled. Thank you for the reference! I'll look into standardized testing. May 19, 2016 at 12:59
• @whuber do you happen to have any sources for the literature on testing or possibly other examples I can look into? I've read all the academic papers on testing I could find or that I could access, but I haven't come across any that treat the standardized tests as binary vectors to detect cheating. May 19, 2016 at 15:41