# Finding correlations in a large dataset

My main objective is to find correlations between process parameters and their impact (more specifically: the amount of wear measured) on specific sections of equipment used in the process.

I've obtained a table of about 5,000 unique "production cycles". Each cycle is a a single row, with 9 process parameters and 13 different sections identified within a single piece of equipment. The wear with respect to the previous production cycle has been measured. The process parameters and wear per section are individual columns.

Note: Subject to change, these specific process parameters were chosen because their global impact on the wear of the equipment is known, based on previous analysis.

So I have 5k rows and 22 columns. I want the first 9 columns to be analyzed for correlations with any of the last 13 columns.

What is the best method of statistical analysis to perform?
The wear measurement values of every section seem to follow a normal distribution (I can provide histograms if needed).

The methods that I have tried so far: Spearman's Rank Correlation: Spearman's Rho correlation matrix

As you can see the correlations are not what I would have hoped them to be, with every value being lower than 0.3, most by quite a large margin.

I have tried other methods: Kendall's Tau A&B, Goodman and Kruskal's Gamma and Pearson's Product-Moment coefficient for linear correlation. These methods gave even lower correlations (can provide screenshots if needed).

Questions:

1. How to interpret the outcome of the Spearman's Rho analysis (or any of the others)?

For example: Can I compare the correlations among each other? Can I say that when a parameter-section point has a value of 0.25 in the correlation matrix, it has a significantly higher impact than a parameter-section point that has a value of 0.1?

1. Am I even using the correct methods? And if not, which method would be more applicable?

[Edit:]

1. Am I going about this the right way and do I just have to accept that my data does not show signs of strong correlations?

[/Edit]

Personal context: I am a computer science student with very little experience in statistics / statistical analysis. I was dropped pretty much cold-feet into this data analysis project. The computer science part of it is really cool and a nice challenge, but I am struggling with the statistical part of it. I am not fishing for an instant solution to this problem: even pointers to some reading/literature in the right direction would be appreciated.

• I've edited for style. Important point: the image you provided isn't of supported type. You need another. Picky point: I've edited correlation factors to correlations throughout. The former is not a phrase statistical people use at all, as a correlation isn't a factor in any sense. (Please check that your meaning is preserved.) – Nick Cox Jan 31 '17 at 16:11
• I can't see why Goodman-Kruskal gamma is relevant here. Which of the various other correlations you could or should use would depend on looking at some scatter plots. Look at lots and lots of graphs is my main advice. – Nick Cox Jan 31 '17 at 16:14
• If you are using R, you can simply plot your entire dataset (i.e. plot(dataset)) and it will create a matrix diagram that will help in seeing the correlations between all of your columns of data. – Tavrock Jan 31 '17 at 16:17
• Although the correlations are not what you expect they might be right. The world may be a noisier place than you think. – mdewey Jan 31 '17 at 16:55
• Your problem reminds me rolling regression and HMM, since your "production cycles" are essentially a sequence. In your case, observations are 13 wear variables and "hidden states" are process parameters. Training your HMM should learn relationship between wear variables and process parameters that then you may try to interpret. – Vladislavs Dovgalecs Jan 31 '17 at 18:21