How to interpret a confusingly large discrepancy between True Negative Rate and the number of Overspecified Models Selected by LASSO

Question

Here it the link to the GitHub Repository for this project. I am going to preface my question by saying that this problem of interpretation I have run into is in the context of me doing my part as a collaborator on a paper for publication for the first time, and I waited a while before coming here to ask for help on this final conceptual puzzle.

I am responsible for running the 3 benchmark variable selection algorithms on a large set of synthetic datasets, calculating their performance metrics, and analyzing and interpreting them. I am currently interpreting the performance metrics of the 1st benchmark, lasso, and they exhibit two things which don't feel like they should be possible at the same time. They have an extremely high True Positive Rate (aka sensitivity) of 0.867, or equivalently, an extremely low False Positive Rate of 0.133, but simultaneously have a very large number of overall models selected which are categorized as Overspecified (221,269 of the 260,000 total models fit on the 260k synthetic datasets), where by overspecified, I mean they are extraneous variable models, which is a regression equation with all true positives included and at least one false positive as well.

Just to make the problem as clear as possible, here is a screenshot of the Workbook I printed the performance metrics out to from R and store them in:

To me, a very high TNR implies a small number of overspecified models by definition, but that's also the way I counted which models are over, under, and correctly specified, by using the TPR, TNR, FPR, and/or FNR I had created as you can see in the following code snippet:

# Number of Underspecified Regression Specifications Selected by LASSO
N_Under = sum( (TPR < 1) & (FPR == 0) )

# Number of Correctly Specified Regressions Selected by LASSO
N_Correct <- sum( (TPR == 1) & (TNR == 1) )

# Overspecified Regression Specifications Selected by LASSO
N_Over = sum( (TPR == 1) & (FPR > 0) )

And the type of relationship I was confident must exist between the TNRs and the number of overspecified models in my performance metrics does in fact exist between the TPRs and the number of underspecified models, i.e. omitted variable models. Because the TPRs are all 1, there are no underspecified models, which is exactly what I would expect.

For the record, when I was writing the R script to run our 1st benchmark (Lasso) that came from and doing the analysis, I asked for help on here or Stack Overflow dozens of times, but never with this part, and I am fully confident I wrote this part of the code correctly, this is a conceptual issue.

Answer 1

Based on your related question, your calculations of TPR, TNR, FNR and FPR are done within each model based on the number of predictors known to be "true" or "false" from the construction of the corresponding synthetic data set. All models have 30 total candidate predictors but only between 3 and 15 predictors specified as "true" in each data set (N2 in your data format "N1-N2-N3-N4.csv"). Note that the FPR calculated that way depends on the number of specified "false" predictors, something that you don't seem to be taking into account.

Your definition of an "overspecified" model is just that it includes 1 or more predictors known to be "false" by construction of the data set. To put that in perspective, the 14.6% "false positive rate" for Line 14 of your spreadsheet (15 each "true" and "false" predictors) it equivalent to only about a 1% chance of including any "false" predictor in the model, as: $(1-(1-0.01)^{15} = 0.14)$. It's thus quite possible to have an "overspecified" model on that basis even if the probability of making an error on any one "false" predictor is low.

Furthermore, as you increase the number of true predictors you make it more difficult to avoid keeping an occasional "false" predictor in the final model. That's illustrated in the plots of coefficient values against the L1 norm values in this answer, based on your data. As I said in that answer:

LASSO can't really be expected to find a "true" model, just one that works well enough based on its "bet on sparsity"

Finally, your all-or-none evaluation of predictor presence/absence doesn't do justice to what LASSO is accomplishing with this synthetic data set. At least in the examples I looked at, the coefficient magnitudes of retained "false" predictors tended to be lower than those of the "true" predictors and thus wouldn't much affect any predictions made from those allegedly "overspecified" models. Evaluation of the L1 norms of coefficients of "true" versus "false" predictors retained in the model could give a better sense of how much this "overspecification" might matter in practice.