2
$\begingroup$

Typically, I see alpha and lambda tuned in elastic net models to minimize cross-validated error. Yet, I have seen a handful of articles by one set of authors where they instead tuned parameters to maximize the cross-validated proportion of observed cases of an outcome (in this article, sexual assault perpetration) among the 5% of participants with highest predicted risk (i.e., top-ventile concentration of risk). I like the intuitive appeal of this for creating risk prediction models.

My questions are:

  1. Does anyone know of any literature on tuning parameters of elastic nets based on concentration of risk instead of cross-validated error?
  2. Does anyone have suggestions for an efficient way to tune alpha and lambda to maximize cross validated proportion of observed cases within the highest ventile of predicted risk in r? What I have below is clunky, and I'm not certain the best way.
# Example    
# Using the Titanic training dataset
train <- na.omit(train[c("Survived", "Sex", "Age", "SibSp", 
           "Parch", "Fare")])
x <- model.matrix(as.factor(Survived) ~ Sex + Age + SibSp + 
                  Parch  + Fare, data=train)[,-1]
y <- train$Survived

# Lists of potential alpha and lambda values
alphalist <- seq(0, 1, by=0.1)
lambda_grid <- seq(0, 3, 0.1)

# Function to get a the lambda and alpha values that produce the 
# highest proportion of an outcome among the 5% of participants    
# with the highest predicted risk
tuning <- sapply(alphalist, function(a){
  for(l in lambda_grid){
    
    # To get cross-validated predicted risk
    fit <- glmnet(x, y, alpha=a, lambda = l, type.measure='auc',   
                  family = "binomial", nfolds=10)
    p <- predict(fit, newx = x, type="response")
    
    # To save a dataframe with the predicted risk in each
    dfprob <- as.data.frame(cbind(Survived=train$Survived, 
            round(p, 2), dec = cut(p, 
      breaks=c(quantile(p,probs = seq(0, 
          1,1/20),na.rm=TRUE)))))
tab <- table(dfprob$Survived, dfprob$dec)
    prop <- prop.table(tab,2)
    a <- a 
    l <- l
    p <- round(prop[2,20]*100, 2)
    df <- data.frame(a=a, l=l, p=p)
    # Return the proportion of cases in the highest ventile for  
    # each combination of alpha and lambda
    print(df)
  }
  
}
)
tuning

Ideally, I would want the output to look something like this, where I have the proportion of cases in the highest ventile of risk for each combination of alpha and lambda

df <- data.frame(round(matrix(runif(11*31), nrow = 31, 
            ncol = 11), 2)*100)
names(df)[1:11] <- paste("a_", seq(0, 1, by=0.1), sep="")
rownames(df)[1:31] <- paste("l_", seq(0, 3, 0.1), sep="")
df

> df
      a_0 a_0.1 a_0.2 a_0.3 a_0.4 a_0.5 a_0.6 a_0.7 a_0.8 a_0.9 a_1
l_0    14    82     5     7    48    69    75    12     6    25  71
l_0.1  27    58    85    99    32    44    86     4    30    52   5
l_0.2  88    80    76    79    71    48    22    39    62    37  24
l_0.3  89    66    74    32    59    52    40    87    85    69  77
l_0.4  47    59    19     0    62    30    16    75    24    45  40
l_0.5   1    88    64     9    39    78    20    22    51    24  81
l_0.6  70   100    19    54    73    41    39    40    27    27  39
l_0.7  70    37    58    35    23    10    95    21    90     8  70
l_0.8  54    90    70   100    98     9    39    56    91    87  19
l_0.9  85    36    18    69    27    41     0    19    96    11  51
l_1     2    44    88    11    70    53    78    79    44    57  26
l_1.1  95    57    51    87     8    41    71    79    43    18  18
l_1.2  83    88     1    34    45    64    16    13    36    11  56
l_1.3  90    32    85    84     6    95    58    61    47    30  47
l_1.4  64     2     3    98    78     6    28    58    47    25  71
l_1.5  33    25    95    24    27    88    51     8    98    34   2
l_1.6   1    84    14    86    80    10    69    78    39    22  76
l_1.7  86    40    96    74    12    86    50    43     9    91  33
l_1.8  86    33    91    23    58    22    70    11     4    57  81
l_1.9  95    19    89     7    16    19    66    99     0    89  48
l_2    90    38    64    34    45    60    69    71    61    58  49
l_2.1  32    83    87    80    56    73    25    59    32    67  81
l_2.2  18    35    11    45    30   100    64    95    15    61  56
l_2.3  80    61    16    91    91    36    97    68    26    15  29
l_2.4  59    45    16    25    79    76     5    13    25    99  60
l_2.5  45    29     7    94    99    29    12    49     2     8  42
l_2.6  34    72    82    22     2    24    40    33    15    18  39
l_2.7  41    14    58    86    56     5    14    24    83     7   5
l_2.8  68    83    39    91    96    66    32    40    66    83  49
l_2.9  75    49     4    34    95    89    27    83    54    93  51
l_3    88    13    80    28    56    37    49    12    53     6  40

UPDATE: Using the Titanic training dataset, I have seen that several values of alpha and lambda can produce the same highest proportion of observed cases within the highest ventile of predicted risk. The values of lambda determined by cv.glmnet for both lambda.min and lambda.1se are two of these, but there are many with the same value.

So, I am now considering a hybrid approach. First, similar to above, determine the range of alpha and lambdas that produce the highest proportion of an outcome in the top ventile of predicted risk in both men and women (in my case and outcomes there may be gendered differences) Second, among those values, chose the parameters with the smallest cross validated risk. Once I have more efficient code in response to the questions above, I will run it on my own data and let people know here the results. It may be that choosing the parameters with the minimum cross-validated error (or 1se) will almost always also produce the highest proportion in the highest ventile of risk. So, the first step may be unnecessary. I will let others know. For now, any advice on more efficient code than below or literature would still be appreciated.

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.