0
$\begingroup$

I've read a lot about Cross Validation to estimate prediction error, specifically for selecting the number of components in a PCA model (I'm not doing SVD/PCA, but it's very similar), but I can't find much literature on using CV for really, really large sparse data. Could anyone help me out with paper links, personal experience, or other techniques for estimating prediction error?

Also, are there any special concerns for performing CV on Poisson type data?

$\endgroup$
1
$\begingroup$

(Is the data still sparse after the SVD/PCA-like transformation? - It would not be after SVD/PCA...)

In my experience shortcuts (such as the leave-one-out hat-matrix calculations) very soon break down (for me e.g. because I have repeated measurements), and I have to revert to the "real" CV. As cross-validation is embarrassingly parallel and you can choose how many cases should be used for training or testing, I don't really see the point in trying any shortcuts in the CV part.

  • As long as your sparse data format allows easy splitting of cases, you are fine. If it doesn't, you probably also need to explain why this data format is good for applying the model (where typically also single cases or batches of cases are evaluated).
  • I'd first try to use the sparsity in the training. That would be part of the model.
  • Next step would be to use the sparsity for prediction. That would be part of the prediction routines for the model.

  • I'd leave the CV alone: it is IMHO important to be able to easily show (and even check during the CV!) that the CV is done properly. One part of this is to make clear that training and prediction work as black box functions along the lines of

    model = f (sparse training data)
    prediction = g (model, sparse test data)
    

concerns for performing CV on Poisson type data?

I don't think so. CV is completely independent of what distribution your variates follow.

If the Poisson distribution means that certain types of cases are very seldom, you may need to think about which groups of cases you pool and how exactly (e.g. it may be important to report the performance for certain types of rare events separately).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.