I've read a lot about Cross Validation to estimate prediction error, specifically for selecting the number of components in a PCA model (I'm not doing SVD/PCA, but it's very similar), but I can't find much literature on using CV for really, really large sparse data. Could anyone help me out with paper links, personal experience, or other techniques for estimating prediction error?
Also, are there any special concerns for performing CV on Poisson type data?
(Is the data still sparse after the SVD/PCA-like transformation? - It would not be after SVD/PCA...)
In my experience shortcuts (such as the leave-one-out hat-matrix calculations) very soon break down (for me e.g. because I have repeated measurements), and I have to revert to the "real" CV. As cross-validation is embarrassingly parallel and you can choose how many cases should be used for training or testing, I don't really see the point in trying any shortcuts in the CV part.
Next step would be to use the sparsity for prediction. That would be part of the prediction routines for the model.
I'd leave the CV alone: it is IMHO important to be able to easily show (and even check during the CV!) that the CV is done properly. One part of this is to make clear that training and prediction work as black box functions along the lines of
model = f (sparse training data)
prediction = g (model, sparse test data)
concerns for performing CV on Poisson type data?
I don't think so. CV is completely independent of what distribution your variates follow.
If the Poisson distribution means that certain types of cases are very seldom, you may need to think about which groups of cases you pool and how exactly (e.g. it may be important to report the performance for certain types of rare events separately).
