Does the opposite of nested cross-validation make sense? I'm asking the question from a machine learning point of view. I have a dataset with relatively high sparsity, so if I use nested cross-validation for my feature tuning and evaluation, that is tune using CV and test on a separate test set after, I get quite different results if I re-sample the datasets (keeping the proportions) and repeat the tuning process. I wouldn't be bothered by some difference, but in my case the differences cannot be ignored.
I want to try the opposite configuration in which I train on X, and test on Y during development, and then evaluate final results using CV on X+Z where Z is of the same size as Y. Or even simply have X to train during development and Y to test, and then CV on X for final evaluation. Is this cheating? Has someone done this before (references would be useful)? Does it make sense?
 A: 
 two splits, X-Y and X-Y-Z, both summing to the whole data. In the case where there is a Z, I thought it might make sense to have the same training and testing proportions during the development and the evaluation (dev: traininging on X testing on Y; final evaluation: CV on X+Z, training on X-sized sample from X+Z and testing on the rest, which is the size of Y)

While training on X also for the evaluation with Z is obviously closer to the settings during X-Y development, the advantage of using X+Y for training the final model (which is tested by Z) is that X+Y are more cases, thus maybe a slightly better model can be built. You could also see this as a (weak?) counter measure to the risk of overfitting during the X-Y development process.

 have X to train during development and Y to test, and then CV on X for final evaluation

This one is cheating: during the cross validation you'd test on cases that have actually entered the model (somewhat indirectly, but still: those cases were used for development)

I get quite different results if I re-sample the datasets (keeping the proportions) and repeat the tuning process.

This means your models are unstable. You should either restrict the complexity or maybe use variance reduction techniques like bagging / ensemble models.
