6
$\begingroup$

I have a big data problem with a large number of predictors and a non-negative response (time until inspection). For a full model I would use a glm with Gamma distributed response (link="log").

However I would like to find a small model. The "best subset glm" approach does not work for me as I run out of memory - it seems that it is not efficient enough for my setting (big data, weak computer).

So I switched to the LASSO approach (using R packages lars or glmnet). glmnet even offers some distribution families besides the Gaussian but not the Gamma family. How can I do a lasso regularization for a glm with Gamma distributed response in R? Could it be a Cox-model (Cox net) for modelling some kind of waiting time?

EDIT: As my data consists of all data points with the information about the time since the last inspection it really seems appropriate to apply a COX model. Putting data in the right format (as Surv does) and calling glmnet with family="cox" could do the job in my case of "waiting times" or survival analysis. In my data all data points "died" and the Cox model allows to analyse which ones "died" sooner. It seems as if in this case family="gamma" is not needed. Comments are very welcome.

$\endgroup$
1
$\begingroup$

Check out the machine learning application H2O (http://www.h2o.ai/), which can be run from within R. This allows you to fit penalised glms (e.g. ridge and lasso models) with a gamma error and log link (as well as several other error structures).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.