Although I have visited this site several times, this is the first time I make a question, so be kind if it is not in a appropriate form.
My problem is part statistical and part R. I am trying to build a Cox PH model in order to make prediction of unemployment. I have a big dataset, N=32538 with covariates p=37 . I split this sample in 3 parts according to Hastie & Tibshirani, train =50%, test=25% and validation = 25%. So, I now have a training set of N=16270 cases. I would like to reduce the number of predictors, but from what I know and have read, it is not wise to do any kind of stepwise elimination. Therefore, I am trying to perform a penalized cox regression, especially with LASSO, using the R package 'penalized'.
library( penalized )
However, it seems that it cannot run...I am not sure why, but I suppose that either my laptop is not very powerful, or that the respective functions are not very efficient for such a bog dataset.
optL1( Surv ( time, status ) ~ . , minlambda=5, fold=3, data=mydata ) optL1( Surv ( time, status ) ~ . , minlambda=5,maxlamda=15, fold=3, data=mydata )
As you can see, I specify minlambda in the first case and both min and max lambda in the second. If I leave it unspecified, it just crushes my whole OS. Now, my pc runs veeeeeeery slow, and after 3 hours ( the most I left it running ), although it seems still running, nothing at all was produced. Those familiar with this function, know that while it is running, it produces in the console "what is going on". That is , for every lambda that it checks, it shows it in the consore along with the according cvl( log-lik ).
In some cases, but not always, it produces the well-known irritating message of memory error....
Error: cannot allocate vector of size xxx Mb
My details :
session(info) R version 3.2.4 Revised ( 2016-03-16 r70336) Platform: x86_64-w64-mingw32/x64 ( 64 bit) Running under: Windows >= 8 x64 ( build 9200)
For now, I tried to run the function in subsets of the full dataset, and I "managed"" to make it run until N=8000( the half sample).
Do you now if I am doing something wrong in running the specific function, or it is an unsolved problem and I have to find another way to proceed ?
Do you know if there are any other packages in R, that can accommodate more efficient the penalized cox regression, and also be capable of making predictions ?
Many thanks!! Giannis
actually, as you can see, I used the classic formula for regression. Meaning, I used the 'dot' in order to include all the predictors in the model. Moreover, as you maybe have guessed, I have many categorical predictors. Do you think that it is better to add the variables all by name in the model, and specify the factors with :
Because by reading the vignette( penalized) , I realized that they use only continuous predictors, leaving out of the model the categorical ones!