1
$\begingroup$

I am bit stuck when it comes to how best to account for repeated measures in a model with a binary outcome.

I am trying to model some data related to the processing of complaints, and whether a complaint was passed on or not. It is a model that will be used for inference on variables that may be related to whether or not a complaint gets passed on.

The data

The outcome variable is a binary one, and is whether or not a complaint gets passed on.

Within the data are individuals who were subject to a complaint, and the dataset covers complaints received over a period of five years. Some of the individuals in the dataset are subject to two or more complaints during the period, but the vast majority are only subject to one complaint only. Where there are repeated measures, the clusters are generally very small (most individuals subject to more than one complaints were only subject to two complaints during the period, but some are subject to 10 or more, and in one case, almost 100).

So, basically there are repeated measures in the data, but not in the majority of cases.

The model

I figured I could either:

  • Just include the first complaint received against any individual during the period in the data for a logistic regression model, and exclude the rest of the data.

  • Run a series of models that only include one randomly chosen complaint against each individual in the data for a logistic regression model, and then average the coefficients and test statistics across the models.

  • Find an appropriate model than accounts for repeated measures in the data, but that does not bias coefficients towards the outcome of cases for individuals that were subject to many complaints. Such individuals appear to be more likely to have a complaint against them passed on.

I'm thinking overall, the second option may be best, but was just wondering whether a better model might be possible that includes all of the data, and also whether the second option would give valid results.

$\endgroup$

1 Answer 1

0
$\begingroup$

Have you tried running a Random Forest model with all of the data? Since each tree is trained in random data subsets with substitution, it could yield quite unbiased results.

From personal experience, you could also try to use partial least squares, where in the k-fold cross validation, instead of being random, you could assign all the complaints about some individual 'x' to the same group; therefore, it would be much less biased (I developed a PLS model where I have a dataset with properties very similar to yours, and the results were very good!)

$\endgroup$
5
  • 1
    $\begingroup$ That doesn't cover the repeated measures aspect. And the sample size needed for RF is astoundingly high, otherwise results may not be reliable/replicated in another dataset. $\endgroup$ Aug 27, 2020 at 11:46
  • $\begingroup$ What is the size of your dataset? Why do you say it doesn't cover the repeated measures aspect? I built a PLS predicted model with repeated measures, and using that specifically cross-validation technique the results were actually unbiased $\endgroup$
    – Johanna
    Aug 27, 2020 at 11:49
  • $\begingroup$ Thanks for your answer and suggestions. I'll look into PLS models. I don't have much experience with them currently. The dataset has around 2100 observations, with around 1800 unique individuals in it. In your analysis, did you include the same individual in one fold, or separate them out to make sure there weren't any repeated measures in the same fold of the data? $\endgroup$
    – Ben M
    Aug 27, 2020 at 12:19
  • 1
    $\begingroup$ That sample size may be quite enough for random forest modeling, actually. The results may not be perfect, but I've seen papers with less than 1000 observations yield pretty good results with random forest modeling (I'm not being able to find the specific paper I'm talking about, but when I do I'll edit the comment to link it) $\endgroup$
    – Johanna
    Aug 27, 2020 at 12:24
  • 2
    $\begingroup$ Beg to differ. You'll find horrendous absolute predictive accuracy and extremely wide confidence intervals for variable importance measures. On the accuracy point always obtain an unbiased full continuous calibration curve using repeated 10-fold CV or bootstrap. You'll often be shocked at how misleading RF is. $\endgroup$ Aug 30, 2020 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.