0
$\begingroup$

I am a novice in statistics and understand more the concepts than what is going on "under the hood", so please excuse and naivety. I am trying to prove whether a R² value for a specific regression model is "statistically significantly" better than another one.

Here is a basic background of the experiment. I will refer to my dependent variable as VAR A, this is the variable that I am trying to model. I have various independent variables that I want to use to model VAR A. I will refer to 2 for the sake of brevity, say VAR B and VAR C are two of my independent variables.

I train a regression model with: (VAR A; VAR B) say experiment 1 and with (VAR A; VAR C) say experiment 2 and each model results in a R² value. Say the results of the two models are: experiment 1: R² = 0.856 experiment 2: R² = 0.834; Using this example I want to say that VAR B is better for modelling VAR A and, either be able to say: that VAR C is significantly weaker than VAR B OR that VAR C is NOT significantly weaker than VAR B (with reference to statistical significance).

To be able to do this I used ANOVA. I am aware than ANOVA will test for significance between two populations, not two numbers. Therefore to create my population I recursively executed experiment 1 and 2 (say a 1000 times), each time the dataset was randomly split into a 70/30 train/test split, where the regression model was trained with 70 % to produce a R² value and the 30 % was used to derive an RMSE of the trained model. After this exercise I had a population of R² and RMSE numbers for both experiment 1 and 2. Using these populations as input to ANOVA it would give me an f and p value which I use to claim statistical significance or not.

Therefore my question is if this is a sound statistical approach? Or am I making some fundamental errors? I hope it make sense, any comment or criticism will be greatly appreciated.

Thank you

Improve this question
$\endgroup$
4
  • $\begingroup$ I'm not 100% clear about what you are using ANOVA for, but a simple way to compare it would be to, instead of making one 70/30 split, you could make a lot (say, 100) of those (all random), and calculate the RMSE on all of them (R^2 is not really ever used in serious model comparison applications), and then conducting something like a Wilcoxon two sample test to test if the RMSE of one approach is larger than the RMSE of the other $\endgroup$
    – Sam
    Commented Oct 19, 2017 at 12:51
  • $\begingroup$ Hi @Sam thanks for the comment. So I am using ANOVA to determine if the my generated R² and RMSE population (n = 1000) from experiment 1 is significantly (p < 0.001) different than the R² and RMSE populations from experiment 2. If the ANOVA result indicate significance (p < 0.001) I make the assumption that VAR B was performed significantly better than VAR C for the modeling of VAR A. Hope that makes sense? $\endgroup$
    – Jascha Muller
    Commented Oct 19, 2017 at 13:08
  • $\begingroup$ How do you calculate this "ANOVA" statistic? $\endgroup$
    – Sam
    Commented Oct 19, 2017 at 13:18
  • $\begingroup$ @Sam my environment is Python (sklearn and scipy libraries), therefore I send the population R² of experiment 1 and experiment 2 as arguments (and separately the RMSE populations) and then receive the F statistic and p value as a result. $\endgroup$
    – Jascha Muller
    Commented Oct 19, 2017 at 13:37

1 Answer 1

Reset to default
0
$\begingroup$

I would say, perform a linear model of your data where you have VAR A as response and VAR B and VAR C as explanatory and compare whether having VAR C and VAR B or VAR B alone (or the opposite) yield a better fit of the model (by comparing them using F or Chi-square tests).

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the response @AP25, so are you suggesting (conceptually) a multivariate regression analysis compared to a single variable regression analysis? $\endgroup$
    – Jascha Muller
    Commented Oct 19, 2017 at 13:11
  • $\begingroup$ If you do a multivariate regression analysis, you should get as output how much each parameter (therefore for each variable) explains your total model. Kinda in that spirit: stats.stackexchange.com/questions/115239/… $\endgroup$
    – Al3xEP
    Commented Oct 19, 2017 at 13:14
  • $\begingroup$ Yea, I understand that, thanks. I additionally run Decision tree and RandomForest multivariate regression models and do that exactly. However, if the one variable scores 50% importance and the other 48% importance, is that statistically significant? I guess I am more struggling with the concept of statistical significance and where to use it. $\endgroup$
    – Jascha Muller
    Commented Oct 19, 2017 at 13:34
  • $\begingroup$ Yes. shouldn't your decision tree algorithm send you back p-values? I'm not familiar with such methods. I guess you can even do single variable regression analysis and compare to a null model for each of your variable using F or Chi-square test $\endgroup$
    – Al3xEP
    Commented Oct 19, 2017 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.