Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.

Let's say I have a list of 100 MLB pitchers and 5 statistics for each of them. The difference between, for example, an ERA of 3.5 and 3.1 might not look like a lot to a naive similarity algorithm, but is a lot in baseball. Given that a lot of the player statistics that I'm looking at have this little variance, a lot of a little variance like this, what is the best way to calculate similarity between two players?

An example of the data might look like this:

Player | ERA | Wins | Strikeouts
--------------------------------
A      | 3.5 | 15   | 180
B      | 3.1 | 12   | 210
C      | 3.4 | 13   | 150

I've used cosine similarity, and the results are all too similar, down to a thousandth of a decimal place.

share|improve this question

1 Answer

up vote 3 down vote accepted

This is not the kind of data you'd use cosine similarity for. Have you considered just averaging z-scores? z-scores will be sensitive to this low variability you talk about.

share|improve this answer
thank you, that is a good suggestion. i'm getting the feeling i let myself get too far away from common sense. – Carson Jan 16 '12 at 21:38
question: even with z-scores, how would i compare all items in each list? – Carson Jan 16 '12 at 21:39
not sure what your question means... maybe make it clearer and part of the main question? – John Jan 16 '12 at 22:14
The z-score approach @John is referring to would be used to compare the values in a column compared to the average of that column. If this doesn't work you could weight the values using the number of games they've played in. Alternatively if you wanted to measure all three columns simultaneously to compare players you could use a multivariate statistic like Hotelling's T^2. Another idea might be to treat it as a 3x3 contingency table and perform an appropriate analysis that would tell you if there is a significant difference, but you'll have to go back and look at the cells to see which ones. – Chris Simokat Jan 17 '12 at 19:46

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.