I am a novice in statistics and understand more the concepts than what is going on "under the hood", so please excuse and naivety. I am trying to prove whether a R² value for a specific regression model is "statistically significantly" better than another one.
Here is a basic background of the experiment. I will refer to my dependent variable as VAR A, this is the variable that I am trying to model. I have various independent variables that I want to use to model VAR A. I will refer to 2 for the sake of brevity, say VAR B and VAR C are two of my independent variables.
I train a regression model with: (VAR A; VAR B) say experiment 1 and with (VAR A; VAR C) say experiment 2 and each model results in a R² value. Say the results of the two models are: experiment 1: R² = 0.856 experiment 2: R² = 0.834; Using this example I want to say that VAR B is better for modelling VAR A and, either be able to say: that VAR C is significantly weaker than VAR B OR that VAR C is NOT significantly weaker than VAR B (with reference to statistical significance).
To be able to do this I used ANOVA. I am aware than ANOVA will test for significance between two populations, not two numbers. Therefore to create my population I recursively executed experiment 1 and 2 (say a 1000 times), each time the dataset was randomly split into a 70/30 train/test split, where the regression model was trained with 70 % to produce a R² value and the 30 % was used to derive an RMSE of the trained model. After this exercise I had a population of R² and RMSE numbers for both experiment 1 and 2. Using these populations as input to ANOVA it would give me an f and p value which I use to claim statistical significance or not.
Therefore my question is if this is a sound statistical approach? Or am I making some fundamental errors? I hope it make sense, any comment or criticism will be greatly appreciated.