There are several posts on here about this question. The gist of them, as far as I understand, is that you cannot compare RMSE or MAE of two models where one is log transformed on the dependent variable since they are on different scales. So you can compare thousands of dollars to thousands of dollars but not log(thousands of dollars) to just thousands of dollars.
I then found this article by Duke What's the bottom line? How to compare models.
In particular, this paragraph:
The root mean squared error and mean absolute error can only be compared between models whose errors are measured in the same units
(e.g., dollars, or constant dollars, or cases of beer sold, or whatever). If one model's errors are adjusted for inflation while those of another or not, or if one model's errors are in absolute units while another's are in logged units, their error measures cannot be directly compared. In such cases, you have to convert the errors of both models into comparable units before computing the various measures. This means converting the forecasts of one model to the same units as those of the other by unlogging or undeflating (or whatever), then subtracting those forecasts from actual values to obtain errors in comparable units, then computing statistics of those errors. You cannot get the same effect by merely unlogging or undeflating the error statistics themselves!
Does this mean that a sound approach to choosing between a log model and it's equivalent non log version would be to do e.g. cross validation and pick the model with the lowest RMSE or MAE after predicting on e.g. 5 folds?
Put another way, the questions I found while researching gave descriptive reasons why you cannot compare RMSE directly with initial model output, but if I understand if I just use the model to predict on test data, I can have a somewhat definitive answer to which model to select?
Is this a logical approach? A standard or typical approach? A good approach?