1
$\begingroup$

Let's say I have a measurement of height of 200 individuals as $y_1$ and fit a simple linear regression model as:

$y_1 = x_1 + x_2 + \epsilon_1$

where $x_1$ and $x_2$ are some subject metadata

I would use $R^2$ or $R^2adjusted$ to quantify the variance of $x_1$ and $x_2$ explained by $y_1$

Let's say I have a new measurement of height of the same 200 individuals (from a different measurement technique) as $y_2$ and a fit a simple linear regression model as:

$y_2 = x_1 + x_2 + \epsilon_2$

Same as above, I would use $R^2$ or $R^2adjusted$ to quantify the variance of $x_1$ and $x_2$ explained by $y_2$

Here, since $x_1$ and $x_2$ are subject metadata they are same in both models.

How should I compare the difference in variance explained by $y_1$ and $y_2$? Simply, I could use the difference in $R^2$ but I am not sure if this approach is correct and also could not think of a proper way to test the significance of the difference in $R^2$.

In other words, how would I pick a measurement method based on the variance (of $x_1$ and $x_2$) explained?

$\endgroup$

1 Answer 1

1
$\begingroup$

First, note that $R^2$ describes the proportion of variance in $y$ explained by the $x$ features.

Getting to your question, I would use an indicator variable $m$ (for “measurement technique”) to indicate which measurement technique was used. Then you wind up with $400$ observations and a regression like:

$$ y=\beta_0+\beta_1x_1+\beta_2x_2+\beta_3m+\epsilon $$

Then you test if the $m$ variable is a significant contributor. This is the $t$-test of the coefficient on $m$, and this is common to have in software outputs.

With $400$ points, you might have enough observations to interact $m$ with the $x$ features, resulting in a regression like:

$$ y=\beta_0 +\beta_1x_1 +\beta_2x_2 +\beta_3m +\beta_4x_1m +\beta_5x_2m +\epsilon $$

A reasonable test in this case would be a “chunk test” of the coefficients that involve $m$, the common one in this situation being a partial $F$-test of nested models with a null model $y=\beta_0+\beta_1x_1+\beta_2x_2$ nested within the above equation.

In fact, the $t$-test also can be seen as a chunk test, with the same null model nested within $y=\beta_0+\beta_1x_1+\beta_2x_2+\beta_3m$.

To measure the effect size, you have the $R^2$ (or adjusted $R^2$) from the model with $m$ (either alone or with interactions) and the model that excludes $m$. Consider their difference.

Throughout all of this, whether you model with $m$ or not, you use all $400$ observations.

In more generality than just your question, understanding chunk tests reveals a lot of beauty in statistics, as many common procedures are special cases of testing nested models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.