0
$\begingroup$

I have been reading a bit on overfitting models, and have come to the conclusion that my regression model is overfitting. Below are the results:

Epoch 1/10
1000/1000 [==============================] - 9s 9ms/step - loss: 7.6314 - val_loss: 15.1603
Epoch 2/10
1000/1000 [==============================] - 9s 9ms/step - loss: 5.7234 - val_loss: 15.1000
Epoch 3/10
1000/1000 [==============================] - 9s 9ms/step - loss: 5.3136 - val_loss: 15.3480
Epoch 4/10
1000/1000 [==============================] - 9s 9ms/step - loss: 4.9874 - val_loss: 15.6923
Epoch 5/10
1000/1000 [==============================] - 9s 9ms/step - loss: 4.7424 - val_loss: 16.2072
Epoch 6/10
1000/1000 [==============================] - 9s 9ms/step - loss: 4.5751 - val_loss: 16.1440
Epoch 7/10
1000/1000 [==============================] - 9s 9ms/step - loss: 4.4371 - val_loss: 16.3838
Epoch 8/10
1000/1000 [==============================] - 9s 9ms/step - loss: 4.3040 - val_loss: 16.3787
Epoch 9/10
1000/1000 [==============================] - 9s 9ms/step - loss: 4.2010 - val_loss: 16.5340
Epoch 10/10
1000/1000 [==============================] - 9s 9ms/step - loss: 4.1154 - val_loss: 16.7138

From much of what I have seen online, there are a few ways to correct for this:

  1. Increase the data size

However, below are my input data sizes:

The input data will be (18114446, 3, 3, 7) for training data
The input data will be (14808983, 3, 3, 7) for validation data

Therefore, I doubt the issue lies here, or might there be a thing as too much data (?)

  1. Reduce model complexity

This seemed like a possibility. However, I tried running a model with only one channel and got the results:

Epoch 1/10
1000/1000 [==============================] - 10s 10ms/step - loss: 12.3651 - val_loss: 6.9210
Epoch 2/10
1000/1000 [==============================] - 9s 9ms/step - loss: 11.7127 - val_loss: 7.3061
Epoch 3/10
1000/1000 [==============================] - 9s 9ms/step - loss: 11.2067 - val_loss: 10.4123
Epoch 4/10
1000/1000 [==============================] - 9s 9ms/step - loss: 10.4911 - val_loss: 12.0211
Epoch 5/10
1000/1000 [==============================] - 9s 9ms/step - loss: 9.9465 - val_loss: 12.2317
Epoch 6/10
1000/1000 [==============================] - 9s 9ms/step - loss: 9.5575 - val_loss: 12.7973
Epoch 7/10
1000/1000 [==============================] - 9s 9ms/step - loss: 9.2371 - val_loss: 12.7207
Epoch 8/10
1000/1000 [==============================] - 9s 9ms/step - loss: 8.9251 - val_loss: 12.1053
Epoch 9/10
1000/1000 [==============================] - 9s 9ms/step - loss: 8.6267 - val_loss: 11.7859
Epoch 10/10
1000/1000 [==============================] - 10s 10ms/step - loss: 8.3835 - val_loss: 11.6321

where the validation data seems to increase right off the bat, and more vigorously than the model where I have 7 channels. Is this also an indication of the model being too complex, or perhaps not complex enough? The general layout of my model looks like the following:

conv2D(64, (3,3), activation='linear')
LeakyRelU(alpha=0.5)
MaxPooling2D((2,2))

Flatten()

Dense(512)
GaussianNoise(0.1)
LeakyRelU(0.5)
Dropout(0.5)

Dense(32)
LeakyRelU(0.5)
Dropout(0.5)

Dense(1)
LeakyRelU(0.5)

optimzer = adam, loss = mae, batch_size = 1000, epoch_size = 10, batch_steps = 1000
  1. Pre-process the data

I have normalized the dataset for each channel (zscore) as well as binned the data (for example, there are lots of instances when low values are present compared to large values) to ensure a near equal representation between low, medium, and high values. In other words, I basically artificially inflated (i.e., copied) the occurrences / instances of rare larger values. I am not sure if there is something else that could / should be done here?

I guess I am at a bit of a loss why the CNN model seems to be performing the way that it is. I used the same data for an LSTM that got decent results, therefore I know it is not the data itself.

$\endgroup$

1 Answer 1

0
$\begingroup$

Tough to say without more details of the problem, but here are some thoughts:

Yes, I agree the first approach is overfitting.

Yes, I agree that you have plenty of observations for just about any application.

I would let model 2 run for more epochs just to confirm it is also overfitting. Maybe there was a weird local min, the val loss is still improving at the end, it's possible it needed more time.

I would try different batch sizes, different dropout rates (although 0.5 is pretty high), and a longer, skinnier architecture, like 5-10 layers of 32 nodes or something. If this runs slowly, you could definitely downsample. Also try smaller batches. Again, this will slow it down, so you could downsample.

Lastly, I'd want to clarify why you are using CNNs at all. Usually 2D CNNs are used on images, is that what you have? Your data shape almost looks like 3x3 pixel images with 7 channels, which I'm guessing is not quite right. With only 3x3 channels, a fully connecting thing might make more sense. I think you are effectively doing that since you are flattening a convolution that is covering your entire 'image'.

Hope that helps some, and good luck!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.