1
$\begingroup$

A subset of my dataset looks as follows where cells in "cat1_ids" column contains list of "cat1" categories and cells in "person_id_list" column contains list of persons id. There are 2000 "cat1" categories and 1500 person, How to prove "no_of_days" is related to to these two columns? After proving a relationship exists the next step is to train a model on this data for predicting "no_of_days". Any suggestions on how to deal with cells containing tuple of values and make a model on it? (both graphical and statistical methods would be appreciated for proving relationship)

  cat1_ids    person_id_list   no_of_days
0   (602,)        (12713,)        1.727083
4   (3, 131)      (12408,)        1.770833
5   (13,)         (12404,)        0.592361
6   (442,)        (12327,)        2.518750
7   (761,)        (12720,)        7.601389

shape => 75000x3

Which model should i use to train over this data to make prediction for no_of_days?

Edit: combinations of cat1_ids has greater influence on no_of_days as compared to person_id_list (it's domain knowledge). Max length of cat_ids and person_id_list is 8.

Improve this question
$\endgroup$
4
  • $\begingroup$ Welcome to CV! Do person IDs occur multiple times? If so, you can model this relationship with a mixed model (specifically, a repeated measures ANOVA). $\endgroup$
    – Frans Rodenburg
    Commented May 9, 2019 at 7:19
  • $\begingroup$ Same person ID must appear several times as well as a category, because there are 1500 person IDs and 2000 categories and 75000 observations (last line of the printout: "format => 75000x3"). You can also change categories and person IDs to frequencies and then try to find a relationship for example: more frequently occurring person has higher no_of_days on average. For prediction, you get person ID, count how many times it occurs in your database and predict no_of_days accordingly. $\endgroup$
    – Marko Lalovic
    Commented May 9, 2019 at 7:50
  • $\begingroup$ Because you have tuples, you could aggregate frequencies in some way (e.g. sum them up) for a start. $\endgroup$
    – Marko Lalovic
    Commented May 9, 2019 at 8:05
  • $\begingroup$ both cat1_ids and person_id appear in multiple cells $\endgroup$
    – dshrikant
    Commented May 9, 2019 at 8:54

2 Answers 2

Reset to default
1
$\begingroup$

Since you have repeated measures, you can model this using a random effect, calculating a random offset from the overall intercept for individuals by estimating their variance. This is simple in the R package lme4:

LMM <- lmer(no_of_days ~ cat1ids + (1 | person_id_list))
summary(LMM)

After that you could bootstrap confidence intervals for the estimates, although that might take a while with that many observations. Note that this model assumes residual normality, so you may need to transform your response variable, or use a different error distribution in the context of a GLMM.

Improve this answer
$\endgroup$
0
$\begingroup$

I would suggest using a repeated measures ANOVA. Like Frans Rodenburg suggested. This has two reasons: 1. it is easy to implement 2. it is designed for related, not independent groups

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.