0
$\begingroup$

I start to study data analysis using R and came across panel data, on the basis of which it is necessary to conduct binary classification.

The data looks like this:

> head(data)
  PERIOD ID V_1 V_2 V_3
1      1  1  27   0   0
2      2  1  19   0  NA
3      3  1  22   0   0
4      1  2  NA  NA   0
5      2  2  28   0   0
6      3  2  27   0   0

The first thing that came to mind was to transform the panel data as follows:

> head(trns)
  ID 1_V_1 1_V_2 1_V_3 2_V_1 2_V_2 2_V_3 3_V_1 3_V_2 3_V_3
1  1    27     0     0    19     0    NA    22     0     0
2  2    NA    NA     0    28     0     0    27     0     0
3  3    26    18     3    26    19     5    28    23     2
4  4    28    30    NA    19    20     1    17    19     0
5  5    28     0     0    25     0     0    30     0     0
6  6    14     0    NA    19     0     3    14    NA     0

But unfortunately, this did not work. It seems to me that this is the wrong approach. Therefore, I turn to the community for advice.

How to work with such data?

UPD1:

Period - period number (consecutive periods, 1 is the oldest);

ID - the customer ID;

V_1-V_3 - customer activity data over the period.

> str(train_target)
'data.frame':   3871 obs. of  2 variables:
 $ ID    : int  1 2 3 4 5 6 7 8 9 10 ...
 $ TARGET: int  0 0 0 0 0 0 1 0 0 0 ...

TARGET - value of the target label (1 - belongs to the segment, 0 - does not belong to the segment).

UPD2:

I added a link to the data.

In addition to tabular_data and train_target described above, the link also has hash values ​​for one categorical variable, which is also desirable to include in the model.

$\endgroup$
4
  • $\begingroup$ Why the NA's? You should join the TARGET variable to the data data frame, and maybe look into logistic regression. Can you share (a link to) the data? $\endgroup$ Commented Dec 22, 2019 at 13:04
  • $\begingroup$ @kjetilbhalvorsen removing missing values that are present in the data leads to a significant reduction in sample size. I think it is not necessary join the TARGET variable to the data. Is not it so? Yes, I can share a link to the data. $\endgroup$
    – Vitalii
    Commented Dec 22, 2019 at 14:36
  • $\begingroup$ When modeling in R, it is a great advantage to have ALL the data in one single data frame, and in so-called LONG FORMAT, as in your data data frame. Please share a link to the data! $\endgroup$ Commented Dec 22, 2019 at 15:24
  • $\begingroup$ @kjetilbhalvorsen TARGET data $\endgroup$
    – Vitalii
    Commented Dec 22, 2019 at 15:28

1 Answer 1

0
$\begingroup$

Since you have given no context for your data, I will give only some ideas and pointers. Using the data from your links, first we need to make one single data frame containing ALL the data. In R:

library(tidyverse)
tab <- read.csv("tabular_data.csv")
target <- read.csv("train_target.csv")
mydata <- left_join(tab, target, key="ID")

Now, since there are missing data NA's, we need to do missing data imputation. That is a large subject, search this site and this CRAN task view. I will use the CRAN package missForest, but it takes a long time, and would certainly benefit from some optimizations.

library(missForest)
mydata.imp <- missForest(mydata)  
  missForest iteration 1 in progress...done!
  missForest iteration 2 in progress...done!
  missForest iteration 3 in progress...done!
  missForest iteration 4 in progress...done!
There were 24 warnings (use warnings() to see them)

On my computer this took almost 2 hours! The object mydata.imp is a list with components

 names(mydata.imp)
[1] "ximp"     "OOBerror"

and the last is of interest

 mydata.imp$OOBerror
    NRMSE 
0.5164205 

which is too high, and indicates that more work have to be done. I leave that for you, start with descriptive statistics, investigate patterns of missingness, and think through which of all these variables are really important.

Then, using the imputed data set, we can do a mixed logistic regression for longitudinal data, using lme4. Note that with all the problems noted above, this is only illustrative, and I include only a few of the variables. A random intercepts model:

library(lme4)
mod0 <- lme4::glmer(TARGET ~ PERIOD + (1 | ID) + V_1+V_2+V_3+V_4+V_5, data=mydata, family=binomial)
 mod0
Generalized linear mixed model fit by maximum likelihood (Laplace
  Approximation) [glmerMod]
 Family: binomial  ( logit )
Formula: TARGET ~ PERIOD + (1 | ID) + V_1 + V_2 + V_3 + V_4 + V_5
   Data: mydata
      AIC       BIC    logLik  deviance  df.resid 
 3652.925  3707.753 -1818.462  3636.925      6991 
Random effects:
 Groups Name        Std.Dev.
 ID     (Intercept) 23.13   
Number of obs: 6999, groups:  ID, 3630
Fixed Effects:
(Intercept)       PERIOD          V_1          V_2          V_3          V_4  
 -33.902330    -0.885390     0.140721    -0.001913    -0.084735    22.966161  
        V_5  
  -1.319042  
convergence code 0; 2 optimizer warnings; 0 lme4 warnings 
$\endgroup$
1
  • 1
    $\begingroup$ Very grateful for your answer. Unfortunately, I do not know the context of this data.I only know that these are some indicators of customer activity over three periods and need to determine which segment an individual client belongs to. I will try to implement your recommendations. I thought of analyzing this data with a method (for example XGBoost) that is resistant to missing values. However, your approach is also quite interesting. The question of including hash values in the model is still open. But this is beyond the scope of this topic. $\endgroup$
    – Vitalii
    Commented Dec 26, 2019 at 9:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.