Holdout set for image task I need to validate whether one or two templates/shapes are present in an image.
Fitting two templates has a better maximum likelihood then fitting one template which is a clear symptom of overfitting. What I would like to do is set aside some test set and then check whether one template or two templates generalise better.
When working with iid data items in regression or classification tasks, we can leave some data at side to use for testing and this can help with model selection. 
(Cross-validation is of course is also an alternative option to keeping a holdout set.)
My question is: can I do something similar for my template problem? I.e. could I randomly  leave some pixels out of training and use them to test whether one template or two templates generalise better? Somehow this doesn't sit well with me as in contrast to the regression/classification problem, here the pixels are not exactly iid (that is neighbouring pixels must be correlated).
 A: I'd say in image analysis the cases to set aside are images not pixels. 
This would apply to cross validation as well as to hold out. 

Update: 
From what you explain about your application, you can and should indeed test on images that were not used for deriving the shape fitting algorithm (i.e. a new set of images). You can then count how many cars were correctly recognized, and how many false positives or false negatives you have.
So this is generalization to unknown images.

Testing left-out pixels: Not only are neighbour pixels not independent but also very basic properties of your image change. Imagine a kind of 4-fold cross validation scheme where you keep every 2nd column and every 2nd row for testing. This corresponds to an image at half the resolution compared to the original image. What you would test with this hold out pixels approach is generalization to (ruggedness agains) lower resolution images.
I.e. this may make sense in certain situations, but you'll measure a completely different performance characteristic.
