2
$\begingroup$

I have build an lstm model to predict human activity recognition with dataset OPPORTUNITY. I did two experiment with different oder of processing as below,

  1. normalized the dataset with minmax scaler, reshape the data to(numberSample,timeSteps(windowlength),numberFeatures), split 70% as training,30% as testing set ,training the model. training and validation loss and acc

  2. split dataset to train,test and validation by selecting data from different person and different trial (follow a paper),then normalized with minmax scaler, reshape to (numberSample,timeSteps(windowlength),numberFeatures), training the model.

training and validation accuracy The dataset is highly imbalanced, I did not set class weight to the first one, but use class weight to the second one. The first one is working better than the second one, the second one is overfitting and shows worse accuracy. What is the reason? does the first one is responsible for Human activity recognition,since I disturb the original order of the sequence?.

Improve this question
$\endgroup$
3
  • $\begingroup$ @Tim the issue doesn't seem to be mostly related to a problem caused by not employing time series cv. $\endgroup$
    – gunes
    Commented Jul 3, 2020 at 14:11
  • $\begingroup$ @ Tim, what is this mean? $\endgroup$
    – Liu
    Commented Jul 5, 2020 at 14:15
  • $\begingroup$ there was a comment under your post and it's now gone $\endgroup$
    – gunes
    Commented Jul 5, 2020 at 14:16

1 Answer 1

Reset to default
1
$\begingroup$

In the first one, there is data leakage between training and test sets. The preprocessing should take place in the training set, and using the fitted preprocessing modules, you should transform your test and validation data. So, the first one is expected to be more optimistic on the performance.

In addition, as far as I understand, you don't need time series splitting because data samples are human actions that are not temporally related. Finally, in general, I urge you to compare any two cases with only one change, e.g. don't change both class weighing and preprocessing step ordering at the same time.

As a side note, accuracy is not a good measure of success, especially for imbalanced datasets.

Improve this answer
$\endgroup$
6
  • $\begingroup$ @ gunes, Thanks form your comment, I agree with this, because I use the whole data to nomalizaition, it means the information from test set alreay be used on training data. I know accuracy is not good, but there is no f1 score can be used as metric in Keras. I evaluate the testing results with the f1 score. What I am not so clear is, "you don't need time series splitting because data samples are human actions that are not temporally related." could you expain in detail. $\endgroup$
    – Liu
    Commented Jul 5, 2020 at 14:11
  • $\begingroup$ With time series data, you typically apply time-series cross validation. But, this is the case when all the samples are temporally correlated, i.e. you do future predictions based on past data. But, in your case, there are different sets of human actions which are not temporally related. $\endgroup$
    – gunes
    Commented Jul 5, 2020 at 14:18
  • $\begingroup$ @Liu is the explanation clear? $\endgroup$
    – gunes
    Commented Jul 6, 2020 at 10:27
  • $\begingroup$ @ gunes Thanks, yes, that is true, so using train test split the data randomly is reasonable for my case. $\endgroup$
    – Liu
    Commented Jul 6, 2020 at 14:37
  • $\begingroup$ where can I accept it? $\endgroup$
    – Liu
    Commented Sep 22, 2020 at 7:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.