Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 1 down vote favorite

We are building a score card system to measure the performance of various groups in the organization. The scores are generated by true / false metrics on individual transactions that each group completes.

Transactions are qualified for measurement by some metrics but not others. The number of transactions also varies between groups. Here is a sample of what the data looks like:

Group  Transaction   Metric1Score  Metric1Qualified   Metric2Score  Metric2Qualified  ... 
G1     Tran1         1             1                  0             1
G1     Tran2         1             1                  1             1
G2     Tran3         0             0                  0             1
...

We'd like to combine all of these transaction metrics into one score per group and be able to compare scores between groups. Does anyone know how to get started on this or have any places to refer to?

share|improve this question

1 Answer

up vote 1 down vote

The first question I'd ask is, what is the business problem you're trying to solve?

The second question I have is - are these groups performing largely the same function? I don't think you can really compare transactional scores unless it's a very similar function; for example, if the groups were the call center reps under each individual supervisor in the center, and the transactions are customer support calls or sales calls. Trying to compare a software engineering group with a financial analyst group in terms of transactional scores is never going to work out - the nature of the transactions and the metrics is far too different.

It's easier to make score comparisons across diverse groups if you have relationship scores, because those are easier to make consistent across groups. Even then there are caveats - a low score is about the relationship, not necessarily the group itself.

I think the best way to integrate diverse metrics is to not do it. At least not analytically. For any business, you have your business strategy and have somehow developed some faith (vision of business leaders, demonstrated analytically, etc.) that certain actions will help you achieve that strategy. Once you have organizational buy-in that something is important, go ahead and measure and report on that thing. If you are solving for reporting up to a higher level of management that doesn't want that level of detail, I think the correct approach is not to try to integrate datapoints that resist integration, but to report at a more abstract level. Report on progress against the strategy and give visibility into how the operating groups are using the transactional metrics, with the opportunity to dive deeper if they are interested.

The "Balanced Scorecard" approach is something I hear talked about in professional groups, although I've never seen it implemented. You can find more information here: http://www.balancedscorecard.org/BSCResources/AbouttheBalancedScorecard/tabid/55/Default.aspx

Hope this helps. Please comment if you're able to add some more information about the business problem you're trying to solve.

share|improve this answer
One additional comment on data structure... I would have "nonqualified" as a value in the metric field, not a separate field. In numeric datasets it is common to use "9" or "99" or "-1" for this purpose. Hopefully you are also using a system that makes it easy to store metadata about the structure. That makes it easier to avoid dummy mistakes that can come up when, for example, you're in a rush, or somebody who is unfamiliar with the data structure needs to analyze some data. It's really not a False if the transaction was simply not evaluated on that metric. – Jonathan Sep 2 '11 at 20:06
Thanks for the feedback. I've given all the business context that I can and that is relevant. The groups are peer groups and are being measured in the same way so it's not an issue to compare across groups. The managers above the groups are asking for a combined score which everyone can use to quickly compare performance. Drill-down data will still be available for those who are interested. The reason that we chose to create a separate column to indicate qualification is to make the scores easy to aggregate in the reporting tool. – Eric Ness Sep 6 '11 at 18:04
How related are the different metrics? I'm thinking in combining these metrics, there are a couple of potential pitfalls to avoid. One is mix change - if one group processes a lot of transactions related to one metric that overall all groups score lower on, their overall score may look worse but it may not be attributable to performance. Another is autocorrelation - if these metrics are highly correlated with each other on a single transaction, then performing a simple weighted average for each metric would favor groups that had transactions that qualified for the most metrics. – Jonathan Sep 6 '11 at 20:25
I think the only way to get to an overall group score that combines the different metrics is to combine them on the transaction level. That's where it gets tricky. Would a simple mean (with a range from 0 to 1) of all the metrics represent the overall quality of the transaction? Probably not. If you look at the total scores for each metric you may find 50% of the values are 1 for metric1, but 75% of the values are 1 for metric2, meaning metric1 either has a higher bar or most teams are just doing it badly today. – Jonathan Sep 6 '11 at 20:30

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.