3
$\begingroup$

Introduction: In comparative studies where we plot two methods, $\left\{M_1,M_2\right\}$ we obtain an explained fraction $R^2$, between those studies. This give us no clue as to which method is better. However, if we have three or more methods, hints as to how good each method is begin to emerge. It would be a great contribution to the literature if we could say something about which method is superior based on those $R^2$ values. The desired $R^2$ values are what the explained fraction would be for pair-wise repeat measures for each method taken separately.

The problem: Suppose we have three methods; $\left\{M_1,M_2,M_3\right\}$ of measuring something. Now suppose that they are correlated $\left\{M_1,M_2\right\}\to R_{1,2}^2,\left\{M_2,M_3\right\}\to R_{2,3}^2,\left\{M_1,M_3\right\}\to R_{1,3}^2$, then there may exist some way of separating effects so that we can obtain $\left\{M_1\right\}\to R_1^2,\left\{M_2\right\}\to R_2^2,\left\{M_3\right\}\to R_3^2$

This is clearer if we put in numbers: Let $R_{1,2}^2=0.7,R_{2,3}^2=0.8,R_{1,3}^2=0.9$, then as $\frac{1}{2} \left(R_{1,3}^2+R_{2,3}^2\right)=0.85$

$\frac{1}{2} \left(R_{1,2}^2+R_{1,3}^2\right)=0.8$

$\frac{1}{2} \left(R_{1,2}^2+R_{2,3}^2\right)=0.75$

This suggests that $R_2^2<R_1^2<R_3^2$, so that some information as to how these quantities relate to each other likely exists. Thus the question is What are $R_1^2,R_2^2$ and $R_3^2$ numerically? Or, failing that How far can we go toward characterizing the relationship between $R_1^2,R_2^2$ and $R_3^2$?

Some thoughts on the question. It is unlikely to be so simple that one can treat $R^2$ as additive explained fractions (as covariance is nonzero, I would guess). If one could then for our example data, we might have

$R_1^2\approx R_{1,2}^2+R_{1,3}^2-R_{2,3}^2=0.8$

$R_2^2\approx R_{1,2}^2-R_{1,3}^2+R_{2,3}^2=0.6$

$R_3^2\approx -R_{1,2}^2+R_{1,3}^2+R_{2,3}^2=1.0$

Possible hint: For error propagation with $n=3$, one obtains for total variance

$\sigma _f^2=2 a_1 a_2 a_3 \sigma _1 \sigma _2 \sigma _3 \left(\frac{\rho _{1,2}}{a_3 \sigma _3}+\frac{\rho _{1,3}}{a_2 \sigma _2}+\frac{\rho _{2,3}}{a_1 \sigma _1}\right)+\left(a_1^2 \sigma _1^2+a_2^2 \sigma _2^2+a_3^2 \sigma _3^2\right)$

where the first term of the two part sum represents covariance, and the second variance and where $R$ in my notation is $\rho$.

Right now, I find nothing in the literature on this subject. Although there has been a paper on the confidence interval of an $R_{1,2},R_{1,3}$ difference.

$\endgroup$
6
  • 1
    $\begingroup$ If I understand your question correctly, then the inputs are correlations between pairs of predictors? And the desired outputs are correlations between each predictor and the true data (the "something" that is measured)? In this case the input alone is insufficient to constrain the output, as you have no information on predictor quality. For example, if the 3 predictors are all highly correlated, then they could all be good or all be bad ... but which one cannot be said. (Note: This is really more of a clarifying question. However, I do not have enough reputation to comment, so I am putt $\endgroup$
    – GeoMatt22
    Aug 16, 2016 at 1:37
  • $\begingroup$ The inputs are models, for example systolic blood pressure measured on three different machines. The outputs are the explained fractions that each machine would have if each patient were measured twice on it. However, there are no actual repeat measurements and that is rather the point. Actually, the naive formulas I put in appear to be approximately correct. $\endgroup$
    – Carl
    Aug 16, 2016 at 2:05
  • $\begingroup$ So the desired output is a repeatability for each model, rather than an accuracy? I had assumed "how good each method is" meant the latter. Is the idea that each model is unbiased, so high repeatability implies high accuracy? $\endgroup$
    – GeoMatt22
    Aug 16, 2016 at 2:25
  • $\begingroup$ The answer is more precision than accuracy. One can have a very high $R^2$ values occur between inaccurate measurements. For example, if I added 100 mm-Hg to all the blood pressures, all three would become inaccurate, but the correlations would not be altered at all. It never occurred to me avoid giving the impression of accuracy when citing an $R$ value, one cannot get there from here. $R$-values are normalized covariances. $\endgroup$
    – Carl
    Aug 16, 2016 at 2:49
  • $\begingroup$ OK. I had interpreted the single-subscript $R$ values as correlations between a model and "truth" (e.g. if calibrating a given machine). $\endgroup$
    – GeoMatt22
    Aug 16, 2016 at 3:12

1 Answer 1

2
$\begingroup$

I think I understand your question now, and the following should be closer to what you are after. To start, I will use the example from your comment.

Assume we have a set of machines that measure blood pressure, where a measurement from machine $k$ is given by

$$M_k[x] = a_kx + b_k + \sigma_k\epsilon$$

where $x$ is the (unknown) blood pressure, $(a_k,b_k,\sigma_k)$ are machine-dependent constants, and $\epsilon$ is "noise".

For simplicity, I will ignore finite-sample effects. The standard "regression" assumptions are then $$\langle\epsilon\rangle=0, \sigma_{x,\epsilon}=0, \sigma_{\epsilon_i\epsilon_j}=\delta_{i,j}$$ where angle brackets denote an average and $\sigma$'s are covariances.

Then we have

$$R_k = \frac{a_k^2\sigma_x^2}{a_k^2\sigma_x^2+\sigma_k^2}$$

and

$$R_{ij}^2 = \frac{a_i^2a_j^2\sigma_x^4}{(a_i^2\sigma_x^2+\sigma_i^2)(a_j^2\sigma_x^2+\sigma_j^2)} = R_iR_j$$

Now if we have three machines $M_1,M_2,M_3$, and $R_{ij}^4 = R_i^2R_j^2$ for any pair of them, this implies

$$R_k^4 = \frac{R_{ik}^4R_{jk}^4}{R_{ij}^4}$$

for any distinct $i,j,k$.

So expanding out, and using symmetry of $R_{ij}$, the answer is then

$$R_1^4 = \frac{R_{12}^4R_{13}^4}{R_{23}^4}, R_2^4 = \frac{R_{12}^4R_{23}^4}{R_{13}^4}, R_3^4 = \frac{R_{13}^4R_{23}^4}{R_{12}^4}$$

I checked this out numerically for finite samples, and it seems to hold reasonably (i.e. small error for large sample sizes).

Hope this helps!

$\endgroup$
1
  • 1
    $\begingroup$ Works better than my naive estimate most of the time. I note that the lowest $R$-value can have higher error, but the average error is ~0. $\endgroup$
    – Carl
    Aug 18, 2016 at 0:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.