I am using keras
to train a 5 layer regression model to predict 1000 different thermometers. I train a model and then ask it to predict what the reading will be based on 20 other instruments. I wanted to see if I am doing things correctly, so I trained 10 different models on the same data and with the same parameters and got 10 different model files. I then asked each of the 10 model files to predict a bunch of temperatures. For some of the thermometers, the temperature predictions were pretty consistent across the models, while for others the temperature predictions were all over the place. Here is an example of what I mean:
| predicted_temp_for_sensor_1 | predicted_temp_for_sensor_2 |
|-----------------------------|-----------------------------|
model_0 | 99.1 | 78.1 |
model_1 | 97.2 | 85.5 |
model_2 | 96.1 | 110.7 |
model_3 | 95.3 | 80.8 |
model_4 | 96.4 | 90.8 |
model_5 | 97.8 | 95.7 |
model_6 | 98.6 | 92.5 |
model_7 | 97.9 | 87.1 |
model_8 | 99.4 | 98.8 |
model_9 | 96.1 | 85.6 |
To make it more obvious, here as some summary stats for the two sensors:
| predicted_temp_for_sensor_1 | predicted_temp_for_sensor_2 |
|-----------------------------|-----------------------------|
predictions | 10 | 10 |
mean | 97.4 | 90.6 |
standard dev | 1.3 | 9.0 |
min | 95.3 | 78.1 |
max | 99.4 | 110.7 |
What can this mean? Is this a sign of model uncertainty? Should I interpret this as low confidence in the predictions? I have no idea what to think about this.