3
$\begingroup$

Although I have visited this site several times, this is the first time I make a question, so be kind if it is not in a appropriate form.

My problem is part statistical and part R. I am trying to build a Cox PH model in order to make prediction of unemployment. I have a big dataset, N=32538 with covariates p=37 . I split this sample in 3 parts according to Hastie & Tibshirani, train =50%, test=25% and validation = 25%. So, I now have a training set of N=16270 cases. I would like to reduce the number of predictors, but from what I know and have read, it is not wise to do any kind of stepwise elimination. Therefore, I am trying to perform a penalized cox regression, especially with LASSO, using the R package 'penalized'. 

library( penalized )

However, it seems that it cannot run...I am not sure why, but I suppose that either my laptop is not very powerful, or that the respective functions are not very efficient for such a bog dataset. 

optL1( Surv ( time, status ) ~ . , minlambda=5, fold=3, data=mydata )
optL1( Surv ( time, status ) ~ . , minlambda=5,maxlamda=15, fold=3, data=mydata )

As you can see, I specify minlambda in the first case and both min and max lambda in the second. If I leave it unspecified, it just crushes my whole OS. Now, my pc runs veeeeeeery slow, and after 3 hours ( the most I left it running ), although it seems still running, nothing at all was produced. Those familiar with this function, know that while it is running, it produces in the console "what is going on". That is , for every lambda that it checks, it shows it in the consore along with the according cvl( log-lik ).

In some cases, but not always, it produces the well-known irritating message of memory error....

 Error: cannot allocate vector of size xxx Mb

My details :

session(info)
R version 3.2.4 Revised  ( 2016-03-16 r70336)
Platform: x86_64-w64-mingw32/x64 ( 64 bit)
Running under: Windows >= 8 x64 ( build 9200)

For now, I tried to run the function in subsets of the full dataset, and I "managed"" to make it run until N=8000( the half sample).

Question 1:

Do you now if I am doing something wrong in running the specific function, or it is an unsolved problem and I have to find another way to proceed ?

Question 2

Do you know if there are any other packages in R, that can accommodate more efficient the penalized cox regression, and also be capable of making predictions ?

Many thanks!! Giannis

EDIT

actually, as you can see, I used the classic formula for regression. Meaning, I used the 'dot' in order to include all the predictors in the model. Moreover, as you maybe have guessed, I have many categorical predictors. Do you think that it is better to add the variables all by name in the model, and specify the factors with :

factor(var1)

????

Because by reading the vignette( penalized) , I realized that they use only continuous predictors, leaving out of the model the categorical ones!

Improve this question
$\endgroup$
3
  • $\begingroup$ You're trying to find a good penalty factor by searching over models: it might be an idea to start by seeing how long it takes to fit a single model with a given penalty factor. (Also, is the full model over-fitting in any case? - why do you want to reduce the number of predictors?) $\endgroup$
    – Scortchi ♦
    Commented Mar 24, 2016 at 13:38
  • $\begingroup$ I tried to use only 1 value of lambda with penalized(..., lambda=x ) . But again, it is efficient until N=12000. I have it running now at the full dataset, but does not seem to respond... I want to reduce them basically because I was told to do so, and secondly, as far as I know, 37 predictors are far too many for a cox regression.....Thanks!!! $\endgroup$
    – GiannisZ
    Commented Mar 24, 2016 at 15:05
  • 1
    $\begingroup$ If you have enough events, there should be no problem with only 37 predictors. Just 740 events in your 32538 cases would give 20 events per predictor, a rule of thumb to avoid overfitting in Cox models. If I understand your data set correctly, that would only be about a 2% incidence of unemployment events; if you have even more events, all the better. Limited backwards step-down selection from your full model can be OK if you require parsimony and accept the tradeoff in accuracy; see page 131 of Harrell's rms course notes. $\endgroup$
    – EdM
    Commented Mar 24, 2016 at 17:28

1 Answer 1

Reset to default
1
$\begingroup$

Question 1: I had a similar problem (memory allocation issue with a dataset of 200K rows and 100 features) and contacted the author of the package. He mentioned he is aware of this issue but doesn't have a fix for now. The package is not designed to work on large datasets (just high dimensional datasets).

Question 2: Coxnet, the equivalent of glmnet for cox regression should be able to do what you want. It doesn't work for time-dependent covariates though. Also, you might consider going into subset selection as an alternative.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.