What's the tractable data size for Sparse PCA or LASSO? I'm up to perform certain kinds of sparse decomposition methods on my dataset. However, I'm not sure: what's the tractable data size for the Sparse Decomposition methods?
The dataset is a $10^3\times10^5$ binary matrix, and the expected methods are Sparse PCA, which aims to find a decomposition with minimized coordinate L1-norm and positive constraint. 
I found these things useful:
penalized package for R language. 
nsprcomp pacakge for R
 A: It depends on how you structure the problem.  If you store the data as a sparse matrix, your datasets can get pretty large before you have a problem.  e.g:
#Adapted from:
#http://stat.ethz.ch/R-manual/R-devel/library/Matrix/html/sparseMatrix.html
library(Matrix)
nrow <- 1e3
ncol <- 1e5
nnz <- nrow*ncol*.25
M1 <- sparseMatrix(i = sample(nrow, nnz, replace = TRUE),
                 j = sample(ncol, nnz, replace = TRUE),
                 dims = c(nrow, ncol))

However, the tradoff is that there's no sparse methods for PCA and sparse PCA that I know of.  You can do SVD using irlba, which is pretty closely related to PCA:
library(irlba)
SVD <- irlba(M1, nv=25, nu=0) #Takes a long time
PCs <- M1 %*% SVD$v
head(PCs) #Not true principle components, but close

You can also do lasoo regression using glmnet:
library(glmnet)
betas <- runif(ncol)
betas[sample(ncol, ncol-50)] <- 0
betas[betas!=0]
Y <- as.numeric(M1 %*% betas + runif(nrow))/10
model <- cv.glmnet(M1, Y, alpha=1)
CF <- coef(model, model$lambda.1se)
as.numeric(CF[CF!=0])

However, if you truly want to do sparse PCA, you'll have to use one of the packages you referenced (or something similar), which require dense matrices, and which will probably choke on a 1e3 * 1e5 matrix.
A: As mentioned the glmnet package can handle datasets with thousands of variables without too many problems.
I just tried doing a PCA on a 1e3 * 1e5 matrix in base R, and it actually worked! Took some time, though (about 5 minutes). Memory usage never went above 3.5GB. This would be a good time to install a multithreaded BLAS, if you'll be doing these analyses frequently.
