0
$\begingroup$

I am performing Binary Classification task with LSTM’s.

Data_size (205, 100, 4) - Out of 205 samples 110 belongs to class 0 & 95 belongs to Class 1, all data were shuffled inside a list.

train_test_split : (train : 85 % & test : 15 % , random_seed = 7)
Fixed train data shape = (174,100,6)
Fixed test Data = (31,100,6)

MODEL TRAINING

I trained my LSTM model

1.1) for model Structure see here

1.2) for Acc & Loss graph (both train & Validate) see here – No Overfitting observed

1.3) Prediction result : 3 out of 31 testing data were wrong. (91 % correct prediction)

1st STRATIFIED K FOLD CROSS VALIDATION (CV)

2.1) Fold – 7 ; random seed = 7 ; data_size = (205,100,6)

2.2) Plot acc & loss graph for every folds. Some folds results in Over fitting see here

2.3) Calculated accuracy for every fold and mean acc value is 79.46 % (+/- 5.60 %)

2nd STRATIFIED K fold Cross Validation (CV)

Changed the model structure by reducing the units/neurons in the layers and performed 7 fold CV again with same data size

3.1) Acc and loss plot. Only some folds were over fitted see here

3.2) Calculated accuracy for every fold and mean acc value is 79.00 % (+/- 3.81 %)

Questions

Q.1) Is that fine to chose 7 folds arbitrarily or should i go for 10 folds ? Will that make any difference ?

Q.2) Is there any limit, if my mean acc value should be > than some %, is considered as a good performing model where the hyper parameters chosen is the best ?

Q.3) On comparing 1st and 2nd CV results, 2nd CV results outperforms 1st CV in std deviation but my mean acc is almost common. Also, prediction accuracy is good for model 1 than model 2. In that case, which model hyper parameters should i fix as final ?

Q.4) My friend suggested me to try LOOCV, but will that make any difference ?

$\endgroup$