I'd like to be able to determine a) how much and b) why a non-random sample “differs” from the population (there's probably a better word than “differs”; I was tempted to use “does not represent” but since “representative” has many different possible meanings, I stuck with mine). In other words, given any sample (not random) from a population described by a few continuous variables (for which I have complete data), I'd like to answer two things:
a) How much is this sample “unique” when compared to the whole population? This measure doesn't need to make sense by itself, but it has to be comparable.
b) What makes it unique? From every variable available, which ones of them could be said to “describe” this sample better, and how much for each?
I'll try to explain it better with an example:
Suppose a set of 100 different toys with three different features: color, shape and size, scattered randomly on the floor of a room. I then ask a child to organize these toys in a way that the most similar will be closer together, while the less similar will be far from each other. Note that the features are very different and there's no obvious result; there's not a single arrangement that perfectly clusters/separates the toys in this fashion. Now, after carefully registering the final arrangement (positions of toys), I'd like to pick any subset of toys (mainly groups of toys that are close together) and determine the two measures I described: a) how much the arrangement of this specific subset differs from the whole arrangement; and b) why were these toys put together – because of shape, color, size or (most likely) a combination of each – if so, how much each feature “weighted” on the decision? (Note: in my real cases, the variables are continuous)
Right now, I have been trying to determine these answers by comparing variance. For each variable, I compare its variance inside the sample and outside of it. If the difference is large, this tells me that the sample is very different from the population regarding this specific variable. This also tells me how much each variable “weighted” in the decision of grouping these items – the variance difference for each variable. When the local variance (inside the sample) of a variable is a lot smaller than the population variance, I consider that this sample is better described by this variable; in other words, it's likely that they were put together because they are similar with relation to this specific variable.
However, I feel that this approach is too naive, but I know too little about probability and statistics to understand exactly how and why. My question is: is this problem a known, common problem? How is this usually solved? Is my approach completely off? I don't need to do a very complex analysis on this, just something that gives a reasonably accurate result.