Is is valid to calculate the mean of multiple neural networks results? Recently I've been playing around with Artificial Neural Networks models and I came across a problem that I can't figure out. Essentially, I'm playing around with a model trained on a relatively small sample from which the forecast performance of the trained neural networks model is limited. However, I noticed that if I run the model multiple times, the average of the sample forecast tends to be more satisfying. My question is, is this a valid way to utilize the forecast results? Is there some other way around that I should seek to achieve a similar goal?
 A: Sure! Model averaging is a tried-and-true approach!
In your case, the variation presumably comes from randomly initializing weights at the start of training. You might be able to do even better by using an "ensemble" method. These pool weak classifiers together into produce a stronger classifier that outperforms the best of its constituents.
There are two basic approaches: bagging and boosting. Bagging is already close to what you're doing. Start by training a large set of "sub"-classifiers on different subsets (drawn with replacement from your training data). At test time, the sub-classifiers all vote and the majority (for discrete targets) or average (for continuous) result is returned as the answer. In boosting, a sequence of classifiers are trained. The first classifier weights all of the training data equally. Subsequent classifiers then weight the correctly-classified data less and focus more on those examples misclassified by earlier stages. There are lots of variants of these basic approaches. Most of the bagging and boosting research has used decision trees, but you can swap in any classifier you'd like (assuming you have the CPU for it!).
As an alternative, Geoff Hinton's group has proposed a dropout training procedure for neural nets. Every time a training case is presented, hidden units are randomly (with 50:50 probabilities) omitted from the network and the weights are updated as if they weren't there. At test time, all of the units are enabled, but their weights are halved because twice as many units are enabled. This essentially averages over a large number of possible neural networks.
A: Even better method for forecast ensemble would be to combine forecasts from multiple classes of methods. That could guard against some form of model misspecification and overfitting.  
Last fall I did that in the context of churn analysis from our CRM-system.  
I combined probabilistic classifications from three models and did probability ensemble as a average of three output vectors.  
Of course it might mean that various explanatory variables get very different weighting as compared to original models since ensemblement changes output.  
One option could also be to use weighted ensemblement where weight W_i could be some other than 1/N, where N is number of models. Weights should of course sum to unity.
