I often get different level of responses when I discuss about R-Squared and its relevance to measuring the performance of a Random Forest or GBM model. In general, RMSE is a better and more appropriate measure, but I sometimes encounter cases where the RMSE is acceptable but the R-Squared is way low ~ 0.18 for example in a recent test. How should one interpret this ?
Also, re: tuning parameters in GBM - I use caret in R and as usual I set a range of values for tree (e.g. 1:100), for interaction depth and so on. I need to check the relative influence of the variables once the GBM is run and in different levels of tuning (eg., shrinkage = 0.01 vs shrinkage 0.001) the rel. influence of the variables can become "very" different. Is this indicative of something fundamentally concerning the data or conversely, how does everyone generally address such an issue (i.e, at what point would you stop tuning further given each run takes a long time (large datasets of x 100k rows) and consider a model as with a given set of tuning parameters as final). Although GBM selects the optimal n trees and depth, finer tuning can change that selection, and consequently alter the results of the variable importance. Same question also for Random Forests.
Thanks in advance for your thoughts on this.