0
$\begingroup$

I am developing a two variable multiple regression model. ie. $$ Y = b0 - b1 * X1 + b2 * X2 $$

I am using the following formula for partial F-test from http://luna.cas.usf.edu/~mbrannic/files/regression/Reg2IV.html under the section Testing Incremental R2. The F-statistics calculated is supposed to tell me if adding the second variable is significant (more details in that link).

$$ F= {\frac{(R_L^2 - R_S^2)/(k_L-k_s)}{(1-R_L^2)/(N-k_L-1)}}$$

My first variable has a strong correlation: regression_coeff_string: b1 = 0.664, b0 = 0.035 R2_val: 0.564

My second variable has a weak correlation: regression_coeff_string: b1 = -25.026, b0 = 0.469, R2_val: 0.027

Adding my seond variable only marginally improves the R2 value regression_coeff_string: b0 = 0.0559, b1 = 0.6633, b2 = -5.2222, R2_val: 0.565

However, because I have a sample size 2949, that With $$ R_L^2 = 0.565, R_S^2 = 0.564$$ $k_L$ the number of predictors in the full set being 2, $k_S$ the number of predictors in the subset being 1 $$ F= {\frac{(0.565 - 0.564)/(2-1)}{(1-0.565)/(2949-2-1)}} = 6.77$$

With F(1,2946) at 0.05 confidence having a F_stat of 4.182, the result is significant. But it seems that it is only because the sample size is large. If I sort the second variable X2 in ascending order in Excel and leave the order of the Y and X1 variables unchanged, I would still get a significant F score.

Question: How can I do a fair incremental R2 test for the addition of a new variable in multiple regression when the sample size becomes large?

Simply looking at the R2 of each X variable individually does not take into account that that they may be cross-correlated, that is why I turned to the incremental R2 test to see how the overall R2 improves relative to adding a new variable.

EDIT1:

The context of my example is predicting solar radiation. The first variable is a solar radiation variable from NWP (numerical weather prediction) software (ie. high correlation). The other variables are other NWP output variables and we are trying to improve our prediction.

$\endgroup$
3
$\begingroup$

The test you are doing is "fair", it's just that p-values don't answer the question you want to ask (they often don't). The way to proceed is to figure out what change in effect size is substantively meaningful and base decisions on that.

This is entirely dependent on your field and, indeed, on your question. To illustrate: If 1 in 1000 children misunderstand a question on a test, that is a very small proportion, and won't affect the validity of the test much. But if 1 in 1000 airplane trips end in a crash, that is a very large proportion and would end aviation.

Is there any context in which a change of $R^2$ from 0.564 to 0.565 is important? I can't think of one, offhand, but I haven't had all my coffee :-). Perhaps some variation on the plane crash scenario.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks. I have added context in my post under EDIT1. Right now, an increase of 0.564 to 0.565 is an easy one to answer that it is no good. But with the F-test always returning positive result, and R2 always increase or stays the same, is a R2 increase to 0.6 good? That is why I turn to the partial F-test to check if adding this new variable is worth it. I understand qualitatively your illustration with the test and plane example, can what you are saying be turned into something more formal quantitatively and how can it be applied in my context? $\endgroup$ – frank Oct 23 '13 at 2:01
  • $\begingroup$ The F test doesn't answer the question "is it worth it". It answers "if the real effect in the population from which this sample came was 0, how likely are results as extreme as this". Is it worth it? can only be answered with substantive knowledge. $\endgroup$ – Peter Flom Oct 23 '13 at 10:08
  • $\begingroup$ I see, thanks for the explanation, if you have links on examples of how to apply substantive knowledge to determine if a multiple regression variable is worth it to add, please let me know. $\endgroup$ – frank Oct 24 '13 at 1:00
0
$\begingroup$

You may want to consider other metrics such as adjusted R square, or Mallow's Cp.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks. Let’s say in my example, my R2_val is 0.565 and adjusted R2 val stays the same as 0.565 when I add a new variable due to the large same size. But then how could I tell if adding the new variable is significant (ie. opening question)? en.wikipedia.org/wiki/Coefficient_of_determination The adjusted R2 can be negative, and its value will always be less than or equal to that of R2 . Adjusted R2 does not have the same interpretation as R2—while R2 is a measure of fit, adjusted R2 is instead a comparative measure of suitability of alternative nested sets of explanators. $\endgroup$ – frank Oct 23 '13 at 2:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.