Evaluating proposed under-sampling method I am currently working on an under-sampling procedure to tackle problems that arise when training and test distributions are different.
Does the following experiment set-up enable performance estimation of the proposed technique?

Data sets are first over-sampled, before they are under-sampled. The
  initial data set is then compared to the result set.

Is this experiment on multiple synthetic data sets a valid approach?
 A: Instead of over-sampling and then under-sampling (not sure about that), you could compare the performance of models trained on the training set before and after under-sampling: If train and test distributions are more closely aligned using your method, this might result in better generalization. 
A: Under-sampling, over-sampling and hybrid methods (by combining both) are valid ways to handle unbalanced data, specially on classifiers overly-sensitive to this (such as Neural Networks).
If the train and test distributions are different in the target, by experience I advise using such techniques.
Otherwise, using some data preprocessing techniques can help your model. For numerical features, you can use scalers according to the original distribution of the training set. For categorical features, you can create rules for handling categories in the test set never seen in the train.
Nonetheless, there is nothing like testing the performance of these techniques on some cross-validated training data!
