Biased classification because of data from different sites? Working in neuroscience, we often classify data from different sites. Usually I balance my data for sites - if I have for instance to classify the data for some illness vs. normal health condition, each of the sites the data is recorded at will contribute with an equal number of normal vs. ill data samples (subjects) to the final data set of the two-class classification problem. 
Nevertheless, can the differences from having acquired the data at different sites still bias the classification performance despite the sets being balanced? If so, why?
 A: Given that each site contributes an identical number of normal and ill samples and that each cross-validation test fold is also balanced in this fashion, I don't see a way how having multiple sites can cause an upward (optimistic) performance bias. 
Somewhat trivially, the classification performance is expected to be worse than in the single site case, since using multiple sites adds class-irrelevant variability to your data. The extent of this negative effect depends on the nature and magnitude of the site-related variability and its similarity to the diagnostic signal.
One more consideration - for permutation tests against chance, I would permute labels only within each site, not across sites. If you will use unconstrained label permutation, you will include unbalanced cases which do not really belong to your null distribution.
A: You might be facing a problem of domain adaptation.
It is possible that the sites contains samples that represent different source.
In this case a classifier learnt on one site (or mixture of all sites), might not perform well on some of the sites.
In your case it might be due to a different distribution of gender, age, etc. of the patients considered.
While you can try and cope with the problem by creating a classifier per site, typically the number of samples per site is quite small and this way you won't be able to utilize all your samples.
There are some methods to cope with domain adaptation and I need more details to know which one fits your problem best.
An easy method (as the title suggests) for the problem is described at Frustratingly Easy Domain Adaptation

