I want to estimate a regularized lasso regression model using nested cross validation to determine the best lambda and to get an estimate of internal validity. Specifically I have a binary outcome (event takes place:yes or no) and a large number of predictor variables. I would like to predict outcome. My problem is that my cases are twins and pairs of twins respond similar and score similar on predictor variables. Data are therefore dependent. Using a logistic regression I would use a random effects model to control for the dependency but this is not possible with a regularized logistic regression. My question now is: What is the best way to perform model selection and performance assessment? How would I handle the dependency during (nested) cross-validation?

  • $\begingroup$ Can you please give details of your model? $\endgroup$ – kjetil b halvorsen Jul 9 '18 at 13:52
  • $\begingroup$ It,s a simple logistic regression with an event as outcome (yesor no) and a number of predictors. The problem is that I have 100 twin pairs and data are dependent because twins will have similar predictor and outcome values. $\endgroup$ – Daniel Jul 9 '18 at 21:11
  • 1
    $\begingroup$ Can you please add this new information to the original post? Few people read comments ... and, if you can post (or give alink to) (part of) the data, we can experiment ... My first thought is a mixed logistic model with a random intercept for each twin pair. But there are other possibilities . $\endgroup$ – kjetil b halvorsen Jul 9 '18 at 21:46
  • 2
    $\begingroup$ I don't think that splitting the pairs so one member go to test set, other to trainset is a good idea. That makes impossible estimation of dependency between pair members, and also its testing. Whole pairs should go to either test or train. $\endgroup$ – kjetil b halvorsen Jul 11 '18 at 13:31
  • 1
    $\begingroup$ Regarding the question about how to split the data into a training and test set, the following reference might be of use: Roberts, D. R., Bahn, V. , Ciuti, S. , Boyce, M. S., Elith, J. , Guillera‐Arroita, G. , Hauenstein, S. , Lahoz‐Monfort, J. J., Schröder, B. , Thuiller, W. , Warton, D. I., Wintle, B. A., Hartig, F. and Dormann, C. F. (2017), Cross‐validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40: 913-929. doi:10.1111/ecog.02881 $\endgroup$ – Phil Oct 29 '18 at 9:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.