I have a 70K x 30 dataset and I want to build a regression model on it. Right now, I am running a bunch of algorithms via Weka tool with cross-validation and I compare the RMSE values reported by Weka in order to decide which model works better.
However, after I experiment with Multi layer perceptron, Linear Regression and a bunch of tree-related algorithms, the best performance I got was K-NN algorithm. Since this algorithm is very naive and instance based, I am not sure if just comparing RMSE is the right way.
When experimenting a Regression model, what kind of process should I follow?