Why is it a "statistical sin" to run a large number of correlations and only report the statistically significant ones? I recently read the following statistical 'sin' here:
Something I see a surprising amount in conference papers and even journals is making multiple comparisons (e.g. of bivariate correlations) and then reporting all the p<.05 results as "significant" (ignoring the rightness or wrongness of that for the moment).
Can someone explain this 'sin' to me? I have run about 40 correlation tests and all are significant!
(I presume this sin has something to do with the notion that a statistically significant finding may not necessarily be a meaningful finding.)
 A: If a result is statistically significant at the 95% level, then if you ran 100 tests you would expect to see 5 examples that "pass" the test of statistical significance even if the null hypothesis is true and the effect is due to random chance.  If you perform such a multiple hypothesis test, an adjustment is normally made to compensate for this effect, such as the Bonferroni correction (the Wikipedia page, and links therein,  should give you the information you need).
A: xkcd neatly illustrated the issue of only reporting positive results

A: On a side note: In my opinion running those tests and reporting the results is okay ... BUT it is important to disclose that the results came from just digging around in the data. So the correlations found are not supported by any theory but might be interesting relationships anyway and  subsequent experiments could explore them. Reporting the data also would enable fellow researchers to compare these observations with results from their own studies. In general: yes, report them but make clear what they are.
A: Another side note: Both the Bonferroni and Sidak correction assume independence - although not knowing your data, I strongly assume that your variables (and the relations between them) are not independent. Hence, your power will drop very much as you overcorrect your $\alpha$-level and so the likelihood of detecting "true" correlations is reduced as well.
(If your 40 correlations still all are significant, even after a very conservative correction, I agree with whuber that you probably test something obviuos or trivial...).
Another approach for testing the spuriousness of correlations with a lot of dependent variables could be a randomization approach:
Sherman, R. A., & Funder, D. C. (2009). Evaluating correlations in studies of personality and behavior: Beyond the number of significant findings to be expected by chance. Journal of Research in Personality, 43(6), 1053–1063. doi:10.1016/j.jrp.2009.05.010 PDF
[EDIT: I just found a similar answer by whuber on another thread]
