Suppose we have a set of basketball players, each of which have 9 associated performance categories. Each of these categories has a different distribution. I want to find a good way to represent how far away each player is from the "average player". My method so far has been to calculate the number of standard deviations from the mean for each category (z-score), then take the sum of them. This number is the total number of standard deviations away from the mean across all 9 categories for a single player. Is this number meaningful as a way to measure total deviation from an average player when each of the 9 categories has a different distribution?
By taking z-scores for each category you are normalising a players value within that category. This “normalisation” is common when comparing data across multiple variables. By normalising you are preventing any category that might have a large variance from dominating the resulting sum you propose as a metric.
This sounds like a reasonable strategy but there are pitfalls and further exploration you can do to gain more insights.
One pitfall to look out for is the presence of outliers. If a player is very average apart for in one category where he is outstanding (ie an outlier) then you won’t capture this fine-grained information. Also, outliers may distort the parameter estimation for categories which in turn will affect the normalisation. For example, if one category contains a (not necessarily large) number of outliers then this will inflate the variance. This in turn can result in artificially small normalisation values for the non-outliers. In turn, this category will contribute an artificially small value to the sum.
To counter outliers, you can also use the Median Absolute Deviation as a robust measure of variability within categories that helps protect from outliers
Instead of taking a parametric approach that you describe you could use ranks. Simply add the ranks of a player in each category to get final score. You can rank final scores to work out who is in any percentile. The 50th percentile will be the average player.
Finally, you could run this data through a Principal Component Analysis to get a different view of which players are “close together” and which are ”far apart” (in terms of their direction of variability rather thean in terms of their distance from the average)