I would like to use a hypothetical variant of principal component analysis algorithm that identifies mixes of components not by real-numbered coefficients, but by 0 or 1 coefficients only, i.e., integral mixing instead of fractional mixing.
My goal is to exclude unhelpful variables from a neural network prediction model, thus narrowing the number of predictors. Currently I do this manually and iteratively, using a "variable importance" score using random permutation of individual column vectors (predictor variables) and seeing the effect on predictions. That takes a long time, and I thought a more structured approach would be possible.
Traditional PCA is sort of like L2 regularized -- not really, but metaphorically -- whereas L1 reg is what I need here. I want to know (in advance) the effect on variance when I completely remove a predictor, as opposed to keeping it in but reducing its presence by some percentage.
Is there such a thing?
If not, I guess I am going to use traditional PCA, and try deleting columns that are shown to be lowest contributors to percent variance explained and see what happens. I'd rather, however, have some form of PCA tell me what to expect in percent variance explained, when I remove predictors. And I'm not even sure the relative rankings of predictors would be correct because I'm taking out whole predictors, not merely fractions of predictors while actually leaving them all in the model.
