# How do I determine which set of measurements is better?

I am measuring protein in humans using two different types of measurement techniques, X and Y (measured on different scales). I have two replications for type X and four for type Y. I average the two measurements for X and average the four measurements for Y to get the 'true' protein average. There are about 120 subjects.

There is large variation in the replication measurements and, surprisingly, the relationship between the average X and the average Y values is not very strong (r = .36).

I am trying to determine which measurement technique is better. I would like to use the subject's level of protein as a predictor in a regression model. I have thought of looking at the coefficient of variations and Cronbach's alphas. Should these two results be consistent? Do they apply here? What would you recommend?

Example Data: Update:

Thanks everyone for the thoughtful responses. I've found out more about the data (it was kind of just thrown at me). The protein measurements were done on consecutive days (so X had two days and Y had 4 days). So naturally someones protein varies from day to day which explains the variability in the measurements. The average is then used as an estimate of their current protein intake. Right now this a common approach for measuring a subjects protein level, since there is no way to know what someones current level of protein intake truly is.

I don't like choosing the protein measurement that correlates strongest with the outcome. I don't think the reviewers will like this justification. Looking at various reliability measurements seems like a better approach to me (I prefer the coefficient of variation). I always thought cronbach's alpha is used for reliability of a latent construct (you could make an argument here I suppose that protein is latent).

Also the correlation is indeed low between X and Y. However I computed that correlation ignoring the the measurement error evident in both X and Y. Correcting for this will lead to a stronger correlation. I have also learned that data collection was sloppy, so not having a strong correlation is not surprising.

Even if one or the other has a superior Cronbach's alpha, that won't get to the heart of your question, because you care about validity as a whole, not just that aspect of validity that entails reliability. I'd look for predictive validity. Since a greater amount of error in a measurement will dampen that variable's correlations with others, whichever of the 2 (X or Y) correlates more strongly with your regression outcome is the best choice.

• (+1) Excellent point. This dataset contains no external or objective indicator of true protein level, making it impossible to identify which is "better" in the sense of accurately reflecting the protein level. – whuber Jun 26 '11 at 18:23

Although @rolando2's point is the theoretically right approach (you are interested in the predictive validity of your measurement) using your dependent variable as the criterion for validity analysis seems to be a circular argument in which you take the same data to decide which is the better measure and to estimate the regression coefficients (cf. Vul et al, 2008).

Therefore, also following the tradition of classical test theory, I would recommend you focus on the reliability of each measure. Because, the more reliable a measure the more variance of the true value it can capture.I would go with the Cronbach's alpha and related measures to compare the reliability for both methods. The one to choose is the one with higher reliability.

However, do not forget that more items (i.e., 4 versus 2 measurements) automatically lead to higher Alphas. Therefore, I recommend you have a look at the alpha() function in the psych package for R, which has other measures than just Cronbach's Alpha.

I like @rolando2 's approach, but agree with @henrik about potential circularity. But I propose a different way out of the circle: Split the data into a training set and a test set, and use the first for assessing the measures, then the latter for any further statistical analysis you want to do.

Ideally, you would be able to use some other variable to test the two measurement sets, but you do not mention one.

But before all that, I'd think about what could be going on here. I don't know your field at all, but should two measurements of protein be so poorly correlated? I would have thought that this sort of measurement was better established. Is the data right? Did something get messed up somehow, earlier in the process?

• In general I agree with your idea. However, I think whether one follows my approach or yours depends on the school/field one comes from. If you want to get the best estimate of protein to estimate the regression coefficient for your data, you take my approach (i.e., a classical behavioral science approach). If you want to get the most accurate estimate to predict other data than the one at hand, go with your approach (i.e., machine learning approach). In this case further methods such as bootstrapping could be also beneficial to even get better estimates. – Henrik Jun 28 '11 at 12:51

I don't state I'm confident. But here's what I did with your example data. First, note that the 5 experimental objects (obs) is a sample of population of objects, and so Obs is a random factor responsible for some variation. For each object, you did 2 probes by instrument type X (and, likewise, 4 probes by type Y). Probing is responsible for the rest part of variation (error one). We could estimate these two components of variation, namely, variance of random Obs factor - Var(Obs), - and variance of error Probe factor - Var(Probe) in case of type X and in case of type Y. This can be done by Variance Components Analysis (VCA), one time for X, another time for Y. Compute ratio Var(Probe)/Var(Obs) in both cases. Because in both cases Obs factor is the same (same objects), the ratio will answer the question where the Probe (error) term is relatively stronger, that is, the instrument is worse (less reliable). I did the analysis with your example data in SPSS and it appeared that Y is considerably weaker instrument than X.

• It seems to me that your variance ratio depends on the units of measurement of X and Y and therefore is arbitrary. You have concluded Y is "weaker" simply because its scale has a smaller range than that of X. – whuber Jun 26 '11 at 18:21