Proving statistical significance for regression R² values

I am a novice in statistics and understand more the concepts than what is going on "under the hood", so please excuse and naivety. I am trying to prove whether a R² value for a specific regression model is "statistically significantly" better than another one.

Here is a basic background of the experiment. I will refer to my dependent variable as VAR A, this is the variable that I am trying to model. I have various independent variables that I want to use to model VAR A. I will refer to 2 for the sake of brevity, say VAR B and VAR C are two of my independent variables.

I train a regression model with: (VAR A; VAR B) say experiment 1 and with (VAR A; VAR C) say experiment 2 and each model results in a R² value. Say the results of the two models are: experiment 1: R² = 0.856 experiment 2: R² = 0.834; Using this example I want to say that VAR B is better for modelling VAR A and, either be able to say: that VAR C is significantly weaker than VAR B OR that VAR C is NOT significantly weaker than VAR B (with reference to statistical significance).

To be able to do this I used ANOVA. I am aware than ANOVA will test for significance between two populations, not two numbers. Therefore to create my population I recursively executed experiment 1 and 2 (say a 1000 times), each time the dataset was randomly split into a 70/30 train/test split, where the regression model was trained with 70 % to produce a R² value and the 30 % was used to derive an RMSE of the trained model. After this exercise I had a population of R² and RMSE numbers for both experiment 1 and 2. Using these populations as input to ANOVA it would give me an f and p value which I use to claim statistical significance or not.

Therefore my question is if this is a sound statistical approach? Or am I making some fundamental errors? I hope it make sense, any comment or criticism will be greatly appreciated.

Thank you

• I'm not 100% clear about what you are using ANOVA for, but a simple way to compare it would be to, instead of making one 70/30 split, you could make a lot (say, 100) of those (all random), and calculate the RMSE on all of them (R^2 is not really ever used in serious model comparison applications), and then conducting something like a Wilcoxon two sample test to test if the RMSE of one approach is larger than the RMSE of the other
– Sam
Oct 19, 2017 at 12:51
• Hi @Sam thanks for the comment. So I am using ANOVA to determine if the my generated R² and RMSE population (n = 1000) from experiment 1 is significantly (p < 0.001) different than the R² and RMSE populations from experiment 2. If the ANOVA result indicate significance (p < 0.001) I make the assumption that VAR B was performed significantly better than VAR C for the modeling of VAR A. Hope that makes sense? Oct 19, 2017 at 13:08
• How do you calculate this "ANOVA" statistic?
– Sam
Oct 19, 2017 at 13:18
• @Sam my environment is Python (sklearn and scipy libraries), therefore I send the population R² of experiment 1 and experiment 2 as arguments (and separately the RMSE populations) and then receive the F statistic and p value as a result. Oct 19, 2017 at 13:37

I would say, perform a linear model of your data where you have VAR A as response and VAR B and VAR C as explanatory and compare whether having VAR C and VAR B or VAR B alone (or the opposite) yield a better fit of the model (by comparing them using F or Chi-square tests).

• Thanks for the response @AP25, so are you suggesting (conceptually) a multivariate regression analysis compared to a single variable regression analysis? Oct 19, 2017 at 13:11
• If you do a multivariate regression analysis, you should get as output how much each parameter (therefore for each variable) explains your total model. Kinda in that spirit: stats.stackexchange.com/questions/115239/… Oct 19, 2017 at 13:14
• Yea, I understand that, thanks. I additionally run Decision tree and RandomForest multivariate regression models and do that exactly. However, if the one variable scores 50% importance and the other 48% importance, is that statistically significant? I guess I am more struggling with the concept of statistical significance and where to use it. Oct 19, 2017 at 13:34
• Yes. shouldn't your decision tree algorithm send you back p-values? I'm not familiar with such methods. I guess you can even do single variable regression analysis and compare to a null model for each of your variable using F or Chi-square test Oct 19, 2017 at 14:02