I am trying to build a Random Forests (RF) model using around 2000 observations and a number of features (can be 50 or can 1000, I still do not know which features are to be used).
One way to evaluate the learned model is by plotting true vs. predicted values from the test set. Suppose I observe the following scenario:
On the x-axis we have the true values and on the y-axis the predicted ones. On the left we have the training set and on the right the test set.
For training we can assume the $R^2$ (regression problem) to be around 0.9 while for testing it's around 0.3/0.4. Furthermore, the Mean Absolute Error (MAE) on the test set is around 0.4 while the model predicting just the mean has a MAE of around 0.6 (with values of the target variable between 0 and 5).
My question is then: how can we say whether the model is overfitting or if the chosen features are simply not enough to make the model learn some relation with the target variable? Is it the same problem but stated in a different manner?
One way to avoid observing such a huge difference between training and test sets, is by increasing, for instance, the minimum number of samples required to be at a leaf node. This would decrease the training score to 0.7 and increase the test score to 0.5. But still, what if the problem is with the selected features?