So I came up with this analysis scheme wherein in use a nested cross validation loop, logistic regression with an l1 penalty in the inner loop and then logistic regression w/o the penalty in the outer loop. To break it down:
Step 1. Split data for outer loop into train and test.
Step 2. Using the training data only use logistic regression with cross validation and an l1 penalty.
Step 3. Take the overlap(intersection) of the variables(features) whose coefficients did not shrink to zero over all cross validation folds in step 2.
Step 4. Train a new Logistic Regression model using only the features that survived across all overlaps in step 3 to train a model on the training data, and then perform prediction on the unseen testing data.
Step 5. Repeat until all data has been used for testing in the outer cross validation loop.
I have a couple of questions, I have just started learning about Bayesian analysis and one take away that I've gotten so far is that it involves a reallocation of credibility. Is this analysis considered "Bayesian" because in a sense I am reallocating the credibility by which some features are likely to give better prediction accuracy by using the l1 penalty with logistic regression in the inner loop ? If not, how could I make it Bayesian ?
The second question is, using this method I end up with far better prediction accuracy on the test data than I would just using a nested cross validation scheme and logistic regression (without removing features). Is there any double dipping going on here ? Do you see any statistically unsound uses ?