0
$\begingroup$

This question already has an answer here:

I'm attempting to use a sequence of numbers (of fixed length) in order to predict a binary output (either 1 or 0) using Keras and a recurrent neural network.

Each training example/sequence has 10 timesteps, each containing a vector of 5 numbers, and each training output consists of either a 1 or 0. The ratio of 1s to 0s is around 1:3. There are approximately 100,000 training examples.

I have tried implementing this using Keras, but the loss stops decreasing after the first epoch of training. I've also attempted modifying the hyper-parameters, but to no avail. Is there something I'm missing here?

The training inputs are as follows: (zero padded)

array([[[0. , 0. , 0. , 0. , 0. ], [0. , 0. , 0. , 0. , 0. ], [0. , 0. , 0. , 0. , 0. ], ..., [1.24829336, 0.96461449, 3.35142857, 0.74675 , 0.776075 ], [1.248303 , 0.96427925, 0. , 1.317225 , 1.317225 ], [1.24831488, 0.96409169, 2.74857143, 1.353775 , 1.377825 ]],

   [[0.        , 0.        , 0.        , 0.        , 0.        ],
    [0.        , 0.        , 0.        , 0.        , 0.        ],
    [0.        , 0.        , 0.        , 0.        , 0.        ],
    ...,
    [1.24969672, 0.96336315, 0.        , 1.319725  , 1.319725  ],
    [1.24968077, 0.96331624, 0.        , 1.33535   , 1.33535   ],
    [1.24969598, 0.96330252, 5.01714286, 1.3508    , 1.3947    ]],

   [[0.        , 0.        , 0.        , 0.        , 0.        ],
    [0.        , 0.        , 0.        , 0.        , 0.        ],
    [0.        , 0.        , 0.        , 0.        , 0.        ],
    ...,
    [0.        , 0.        , 0.        , 0.        , 0.        ],
    [1.25715364, 0.95520672, 2.57714286, 1.04565   , 1.0682    ],
    [1.25291274, 0.96879701, 7.76      , 1.311875  , 1.379775  ]],

   ...,

   [[0.        , 0.        , 0.        , 0.        , 0.        ],
    [0.        , 0.        , 0.        , 0.        , 0.        ],
    [0.        , 0.        , 0.        , 0.        , 0.        ],
    ...,
    [1.24791079, 0.96561021, 4.44      , 0.7199    , 0.75875   ],
    [1.25265263, 0.96117379, 2.09714286, 0.7636    , 0.78195   ],
    [1.25868651, 0.96001674, 3.01142857, 1.35235   , 1.3787   ]]])

The training outputs are as follows:

array([[0.],
       [0.],
       [0.],
       ...,
       [1.],
       [0.],
       [0.]])

This is the model I have attempted to train:

#Model 
model = Sequential()
model.add(LSTM(100, input_shape= (10, 5)))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
print(model.summary())
model.fit(X_train, y_train, validation_data = (X_test, y_test), epochs = 100, batch_size = 1000)
$\endgroup$

marked as duplicate by Sycorax, Peter Flom Apr 8 at 11:53

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

1
$\begingroup$

Two things to try:

model.add(LSTM(100, input_shape= (10, 5)))
model.add(Dense(50, activation='relu'))
model.add(Dense(25, activation='relu'))
model.add(Dense(10, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

and try batch_size=32. The default is 16, I think, and 1,000 is pretty large.

I've found that large jumps down in the number of dense units (100 to 1) are more fragile than more moderate jumps. The LSTM cells are creating features, and the Dense layers are creating a classifier, and my guess is you need a better classifier.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.