I am trying to find the prediction interval of a point prediction (via neural network). Basically this is a multiple linear regression problem with several independent variables and one dependent variable. For this I have used multi-layer feedforward neural network (via h2o). The inputs to the network are a few parameters relating to the behavior of an industrial machine and the output is "number of days to failure" of that machine. I have thought of and coded two approaches to find the prediction interval but I am not sure which one is correct (or if there is a third approach)?

Approach 1:

  1. Train the model through a multi-layer feedforward neural network (via h2o) with all the usual steps of building and validating a model.
  2. Implement the model on the latest / most recent available data of that machine (or of a different machine). The model will output one value of "number of days to failure".
  3. Simulate the inputs through randomly changing the values of the input parameters. For example, new value (of input) = original value +/- X% where X is picked up randomly from a uniform distribution within the bounds -X and +X (usually +/- 5%). And the original value comes from an optimization routine.
  4. So simulating 100 times creates 100 rows of input data.
  5. Passing this input data (of 100 rows) through the neural network creates an output of 100 values i.e. 100 values of "number of days to failure".
  6. From this distribution of 100 values, find the 2.5th and 97.5th percentile values to get the 95% prediction interval.

Approach 2:

  1. Same as above
  2. Same as above
  3. Same as above
  4. Same as above
  5. Randomly select N number of rows (N < 100) with replacement B number of times. For instance, randomly select 60 rows (with replacement) 200 times. That is, I now have 200 sets of inputs with 60 rows each.
  6. Pass the first set of inputs (of 60 rows) through the neural network to get 60 values of "number of days to failure". Find the mean of these 60 values.
  7. Similarly pass all the rest of the 199 sets of inputs (of 60 rows each) to end up with 199 mean values for a total of 200 mean values of "number of days to failure"
  8. From this distribution of 200 mean values, find the 2.5th and 97.5th percentile values to get the 95% prediction interval.

Can anyone tell which approach is correct (if at all) and if there is a different approach that is better suited to my problem.

  • $\begingroup$ I may be missing something, but if you first learn your network on some data, and then make predictions on "simulated" data, then the predictions are made using the same "static" (deterministic) model, so it will give the same predictions for the same point, it wouldn't let you estimate any intervals. $\endgroup$ – Tim Jun 1 '18 at 11:09
  • $\begingroup$ Hello Tim, only part of the data used for predictions is simulated, for instance, 1 row of input (out of a total of 100) is coming from actual machine parameters and the rest 99 rows are simulated. So 1 row of, say, 10 variables is sufficient for getting the point prediction. The rest of the 99 rows (of simulated variables) provide a distribution (of point predictions) in order to get the prediction interval. Yes, you are correct that the model will throw the same prediction for the same point always. So can you suggest any approach to tackle this. $\endgroup$ – Ankur Jun 1 '18 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.