I have a model that is based on an experiment collected on 100 subjects. We are testing the model as follows:
- Record raw data from the subjects
- For each subject, compute the feature from the raw data (thus, we obtain 100 datasets with 25 features)
- Reserve the data of one subject for model validation
- Combine the remaining 99 datasets into one large dataset (thus, we obtain a dataset with up to 1 million rows and 25 columns)
- The obtained large dataset is used to train a model and evaluate it using a 10-folds cross-validation.
- Use the reserved dataset (see step 3) to validate the model obtained in step 5 above
Unfortunately, the model's performance is very confusing to me (is this an overfitting? a data leakage?) :
- The cross-validated model achieves 99.9% accuracy and 99.7% recall)
- However, when the same model is tested using the validate test set (see step 3), I get a very low accuracy (40.2% recall and 39.8% precision)
What could be the reason for this discrepancy? Any suggestions on how this could be improved?
NOTE: This question was originally posted here but I was asked to move it to this forum instead.