choosing between traditional statistical models and neural networks using RMSE I am new to neural networks, and am more familiar with classical linear regression type models. I have a basic question about choosing between the two in attempting to develop a predictive model. 
Is root mean squared error a valid approach to selecting a neural network over a regression model, or vice versa, based on the predictive performance? Or, are there other measures by which they could be compared? 
 A: How to select a model depends on what you want to do with the model: a model that's good for one purpose may be bad for another purpose.
RMSE is often used to assess model fit (training error) or, using cross-validation or a separate test set, to assess predictive accuracy (test error). RMSE is no less useful for these tasks with a neural network than with any other kind of data-generating or predictive model, because it only compares model-generated values to observed values, with no concern for the interior structure of the model or how the model uses the predictors.
A: There are a few issues with what you're doing that you must consider, though there's a good chance that RMSE is fine for your purposes.
1) Neural networks have gotten popular in applications like speech and image recognition because they give exceptionally tight accuracy. In exchange for accuracy, neural nets sacrifice the easy interpretation of linear regression where you get p-valuesfor the parameters and can determine which are significantly different from zero and driving the process.
library(MASS)
X <- matrix(rnorm(1000),200,5)
B <- c(1,8,0,-0.1,-3)
e <- rnorm(200)
Y <- X %*% B + e
L <- lm(Y~X)
summary(L)

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.07249    0.06935  -1.045   0.2972    
X1           1.03197    0.06773  15.235   <2e-16 ***
X2           8.07410    0.07320 110.303   <2e-16 ***
X3          -0.03898    0.06436  -0.606   0.5455    
X4          -0.12129    0.06459  -1.878   0.0619 .  
X5          -3.08850    0.06637 -46.535   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

As I rigged it, X1, X2, and X5 are revealed to drive Y, and X4 has a slight influence.
Nothing would prevent a model developer from finding p-values for the parameters of a neural network (e.g. bootstrap or maybe there's an established way to calculate p-values without resampling and retraining), but even if you get them, the interpretation is opaque. Sure, a parameter may be significant, but when it's some nonlinear combination of the input variables, the interpretation is not so obvious to me.
For that reason, neural networks would not be the best choice when your goal is interpretation and inference about why an input is found to have a certain output. Linear regression tends to be the tool of choice. 
2) RMSE might not quite measure what you want. It gives equal penalties to missing high and missing low. In a business setting, those probably are not equally bad types of errors. A metric giving an especially harsh penalty for missing high (or low) may be more appropriate, though I would interpret it as "RMSE-ish".
As Kodiologist mentioned, the appropriate data for calculating any kind of metric would be on out-of-sample data, perhaps some kind of cross validation. Particularly with a neural network, the parameters can start to memorize the training data and essentially play connect the dots, so your in-sample performance is not a reliable measure of how your network has modeled the process that generates your response variable.
Edit: I'm saying "accuracy" because I'm in classification mode. I mean something more like R-squared for a regression.
A: No, RMSE is not a universally best way to measure performance of a model for the purpose of selection of best predictive model even without neural networks in the mix. For example if you are forecasting the likelihood of an event such as death or loan default RMSE is usually not a good measure of performance. 
It can be appropriate in some situations , such as when forecasting revenue of a store. Even here you must be careful if the price of under forecasting is different from price of over forecasting. Imagine you are forecast provisions for a solo trip across the ocean, you probably want to err on side of over forecasting and get too much food rather than too little. In this case instead of RMSE you would want to use a measure that is asymmetrical to the sign of forecasting error
