Model A was fit using 213 predictors, Model B with 252 predictors, on a data set with n=431. I test both on an independent data set (n=121) and am currently using RMSE to determine which model performs better.
I am looking for more measures to compare the predictions of the two models. I looked up mean absolute deviation, but am not sure what exactly what that tells me or if it is useful for a comparison of predictions between both models. Is it the variation of the predictions? Can I use r-squared (r2) or do I need an adjusted r2 since both models have a different number of predictors? I'm not sure how I would calculate the adjusted r2 for the test set because n=121 and p>n for both models. Am i misinterpreting the adjusted r2 calculation?
I have also created a plot of actual vs predicted values. Any suggestions are appreciated.