0
$\begingroup$

I'm playing with a little prediction project using a multi-layer perception (MLP) with robust back propagation. I have a variety of variables which correlate to the single output I'm attempting to predict.

On a whim, I wanted to try providing the neural network with a single input of random uniform values (scaled -1 to 1). The output surprised me.

A simple plot of the target values vs. the predicted values shows that the neural network learned the data to a surprising degree.

The images below describe what I'm observing. The scale of the response makes the plot hard to see, but it should seem obvious that the training and test data perform similarly. I can understand both results having the same distribution and scale, but cannot explain why the test data doesn't show more randomness in it's curve.

The neural network had 125 neurons in a single hidden layer and was iterated 800 times. If anything, I might have expected overfitting on the training data, which should have exacerbated the randomness in the test data's response even more.

I had hoped to use this as a baseline metric to gauge the meaningfulness of other variables in the data set, though I'm not sure how to proceed from here.

I'm sure I've stumbled upon a well-documented phenomenon / technique, here... Any thoughts?

training predictions by sorted targets of neural network using a single random uniform value for input

testing predictions by sorted targets of neural network using a single random uniform value for input

Improve this question
$\endgroup$
6
  • $\begingroup$ What's an MLP? What's r-prop? Substantively, neural networks are prone to overfitting, so the relevant question is how well does the neural network do against data that it hasn't seen before? I suspect the performance will decline significantly when you apply the model to new data. $\endgroup$
    – Sycorax ♦
    Mar 30, 2015 at 20:12
  • $\begingroup$ MLP = multi-layer-perceptron. r-prop is an enhanced algortihm for back-propagation. That's what had me confused though. I plotted an X-Y plot of the target values (ordered from lowest to highest) and the corresponding prediction the NN made on the test data. The predictions tracked rather nicely with the targets. I suppose I could expect the output's distribution to get "trained" into the network, but I wouldn't have expected it to be so consistent with the target values. $\endgroup$
    – Joel Graff
    Mar 30, 2015 at 20:23
  • $\begingroup$ I've made NNET models that perfectly predict the training data! It's best to train neural networks, and statistical models generally, to optimize out-of-sample performance. Usually this is accomplished with repeated $k$-fold cross-validation. $\endgroup$
    – Sycorax ♦
    Mar 30, 2015 at 20:25
  • $\begingroup$ I may try that. Thus far, I've implemented a stratified sampling scheme, using a 70/30 split between the training and test data. The data's sampled randomly without replacement from 4 strata to help ensure consistent sampling across an otherwise long-tailed distribution. Right now, I just need to be able to expose an uncorrelated (or poorly-correlated) input for what it really is. $\endgroup$
    – Joel Graff
    Mar 30, 2015 at 20:40
  • $\begingroup$ Are you now observing correlation on new test data, that were not used for training? $\endgroup$
    – Denis Tarasov
    Mar 31, 2015 at 9:57

2 Answers 2

Reset to default
1
$\begingroup$

It would appear there's something wrong with my current methodology. Haven't figured out what, exactly, but implementing k-fold immediately demonstrated this to me, as a single random input for a value results no meaningful learning occurring in the network.

Back to the drawing board....

Improve this answer
$\endgroup$
0
$\begingroup$

Perhaps your data has duplicate instances, or at least very similar instances, so when taking some instances for testing, you leaving other similar instances in the training data, this situation is as if your are testing and training on the same data. You may use Weka to remove the duplicate instances and do the experiment again and see if you still get the same results.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.