The knock-off method is a recent approach to variable selection and FDR control presented in two papers to be found here https://statweb.stanford.edu/~candes/papers/FDR_regression.pdf and here https://statweb.stanford.edu/~candes/papers/HighDimKnockoffs.pdf. The method has been developed by two professors in the University of Chicago and Stanford and has been implemented in R in two packages: knockoff and MFknockoffs which can be downloaded from CRAN.

Since I want to become familiarized with this method I run the R function on the German Credit Dataset to be found at the UCI Machine Learning Repository (just google it).

However at my surprise the knockoff.filter() function returned an empty vector of selected variables, essentially regarding them all as having zero coefficient. This is counter-intuitive as glmnet --regularized logistic regression using Lasso- through repeated cross validation returns a model with a significant number of variables that performs respectably at the test dataset with AUC = 0.74.

Moreover while the knockoff.filter() returns 8 selected predictors when applied to the mtcars dataset in base R, the corresponding MF.knockoffs.filter() function of the MFknockoffs package returns an empty vector (see below for a reproducible example):

> data(mtcars)
> kf <- knockoff.filter(X = mtcars %>% select(-am), y = mtcars$am, fdr = 0.2, statistic = NULL,
+                 threshold = c("knockoff"), knockoffs = c("equicorrelated"), 
+                 normalize = TRUE, randomize = FALSE)
> names(kf)
[1] "call"      "knockoff"  "statistic" "threshold" "selected" 
> kf$selected
 mpg  cyl disp   hp qsec   vs gear carb 
   1    2    3    4    7    8    9   10 
> result = MFKnockoffs.filter(X = mtcars %>% select(-am), y = mtcars$am)
Warning: doMC is not installed. Without parallelization knockoff statistics will be slower to compute
> names(result)
[1] "call"      "X"         "X_k"       "y"         "statistic" "threshold" "selected" 
> result$selected
named integer(0)

My question begs the explanation of these phenomena. It is not about programming but rather about the application and interpretation of a novel and reputable statistical method of high importance.

Your advice will be appreciated.

  • $\begingroup$ This is not an answer, but at least part of the issue in the example is that the default fdr argument is different between MFKnockoffs.filter and knockoff.filter, with fdr defaulting to 0.2 in the former case and q defaulting to 0.1 in the latter. When run with equal FDR thresholds, MFKnockoffs.filter returns 5 selected variables, rather than zero. Not to diminish the rest of your question, as the methodology is very interesting. What threshold did you use with GCD data set? $\endgroup$ – Chris C Jul 24 '17 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.