2
$\begingroup$

I am running a Random Forest on data which has multiple observations per Subject. Some change with time, most don't.

Subject time  Feature1  Feature2    Feature3  Feature4      Feature5 Feature6   Target
Ann     1     Yr1       Female      20        Special-Yes   3.6      Car    0   
Ann     2     Yr2       Female      21        Special-Yes   4.0      Car    0
Ann     3     Yr3       Female      22        Special-Yes   3.2      Bus    0
Bob     2     Yr1       Male        19        Special-No    2.6      Car    0
Bob     3     Yr2       Male        20        Special-No    2.7      Car    1
Cathy   2     Yr1       Female      24        Special-No    1.6      Bus    1
Diane   3     Yr1       Female      37        Special-Yes   6.6      Bus    1

Feature 3 is age, therefore, increases by 1 each year. Feature 2 and 4 never changes with time. Feature 5 changes over time. Feature 6 may or may not change.

I don't want to model the change in values over time for each subject

All I want to be able to predict if the subject belong to class 0 or 1.

What are the implications of dropping Time and Subject Name and running a Random Forest Model out the features and the target ? Is this valid ? Why not ? Can I consider each row to be independent after dropping time and Subject ?

I can see that there are image classification models that consider multiple images per Subject such as 3 different X-Rays per patient. However, target variable in these cases are consistent across the observations for each Subject, which is not in my data set.

Can I have multiple observations / records per Subject in a classification problem ? If yes, does each record have to be unique to run RF ?

I looked at this post. But couldn't find a convincing answer. My data set looks like this.

$\endgroup$

1 Answer 1

2
$\begingroup$

By construction, random forests heavily overfit on the training set. Thus, never use insample results and predictions for any purpose (except to verify this fact). For independent observations, this can elegantly fixed by studying out-of-bag (OOB) results/predictions. With clustered samples like in your case, OOB predictions are biased and can't help as well. As a solution, I would probably work with cross-validated predictions using grouped sampling. By grouped sampling, I mean that all three rows per patient are kept together in one fold when building the cross-validation folds.

It depends on the aims if it is smart to drop time and subject id. Often, subject id is of no use because the model could not be applied to new patients. Dropping time sounds a bit strange though.

$\endgroup$
1
  • $\begingroup$ Dropping time after splitting the data where time periods 1 and 2 is set aside for training and time period 3 is set aside for testing. However, since the Subject id is dropped, each row for a subject in period 1, 2, and 3 is considered independent. Therefore, a record for the subject in all 3 time periods, could have 2 records in the training set and 1 in the test set. And cross-validation is done randomly as well. I am unable to point exactly what is incorrect in doing so. Is it the data leakage in cross-validation or something else that would make this incorrect ? @Michael M $\endgroup$
    – learner
    Commented Jul 21, 2020 at 0:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.