# Unsure about correlation calculation for a particular setup

Given an experiment where a number of entities are measured/evaluated using two different methods in two different runs, how would one go about calculating the correlation between the runs?

In other words; I have about a thousand entities $X_i$ each with two scores $S_{i,1}$ and $S_{i,2}$, and two such datasets from two runs. I am not in a position to assume anything regarding the real distributions $S_{i,j}$ come from. As mentioned above my goal is to find to what extent these scores correlate between the runs.

Statistics is not my strong suit, so I am not really sure how to proceed but my first instinct is to:

1. Separate the scores from one another and calculate two correlation coefficients for the score models $S_{i,1}$ and $S_{i,2}$.
2. For each entity $X_i$ take the score from first run as x-coordinate and the score from the second run as the y-coordinate, and acquire a scatter plot. Then use the plot as a basis for the calculation of a Pearson's correlation coefficient.

Would this be a decent/acceptable approach? I have read a little bit on Pearson's correlation on wikipedia and calculating the expected values and variances of $S_{i,j}$ feels a bit fishy since I have no clue about the underlying distribution, and particularly whether or not I can assume the scores should come from the same distribution.

In short I am pretty confused, and would really appreciate any help on the matter.

-
As the properties of the Pearson correlation do not make any claims about the distribution of the data, this will not be a problem. Your approach seems valid to me. –  mzuba Feb 8 '12 at 11:16
Are the two scores measuring the same attributes, so they are repeated measures of the same thing? What are your entities? –  Michelle Feb 9 '12 at 4:38
The entities I mention here are networks, and the scores $S_{i,1}$, $S_{i,2}$ are two different models I evaluate them by. The different runs, I believe, can be seen as repeated measures of the same thing since the models are consistent in between the runs. –  posdef Feb 9 '12 at 9:10