1
$\begingroup$

Here's the situation I'm in:

I need to create a measurement to compare different companies by how much they grew in the last year, both in website visits and employees, as part of a larger scoring system that takes many factors into account. Each of the individual scores will get z-scored and the final score is an average of all z-scores.

The issue is that I need to find a way to balance relative and absolute growth in this score. One one hand, a company that went from 100 to 150 employees has grown much more impressively than one that went from 1 to 2, so purely relative growth doesn't work. On the other, a company going from 25 to 100 employees in one year is much more impressive than one that went from 1000 to 1100, so purely absolute growth doesn't cut it either.

It's also not helpful to have huge outliers so this aspect of the score doesn't overshadow others, so some sort of diminishing returns factor would be interesting.

I keep trying different methods to balance the two, but I feel like I'm flying blind. At one point I landed on (relative * absolute^2 )^0.1 which seemed to work okay, but honestly I have little idea if it even makes any sense.

I'm proficient in Python, which I'm working in, but I haven't used much pure math in years, so I'm rusty in my intuitive understanding of what I'm doing here. I keep thinking that maybe I should visualize this function somehow so that I can fine tune the weighting, but I can't quite get it clearly defined in my head how I would do that.

So anyway, that's my problem. Can anyone help shed some light on where I could begin to work or things I should read up on?

Thanks in advance!

Improve this question
$\endgroup$
1
  • $\begingroup$ Scoring as you've described it is not an area with a predetermined, cookbook answer. One approach to dealing with outliers is to use a modified z-score which is more robust to outliers and is described in this link itl.nist.gov/div898/handbook/eda/section3/eda35h.htm. Since you're focused on creating these scores, then adding them together, it's not clear why there is even a question. There are other approaches to combining and scoring metrics such as principal component analysis, factor analysis, correspondence analysis, and so on that might be worth exploring. $\endgroup$
    – user234562
    Commented Apr 1, 2020 at 18:43

1 Answer 1

Reset to default
0
$\begingroup$

When "scoring" a data set, it is essential to have a clear understanding of your optimal scoring method. Is your goal to predict the website views a company gets, or the profit, the probability of failure, or something else? Without a clear goal, you are effectively just playing with numbers until you like what you see.

Once you have identified a clear goal, you formulate that into a fitness function that can allow you to apply your model to your dataset, and get back a single number telling you how well your model fits. From there, you can optimize the parameters of any scoring model you want to try.

If you are after an analytical formula, you are right to simply choose some form that "sounds good" then try them out and see how good they do. Sometimes your best forms will seem a little hard to interpret. This is normal in the field of ML and you have to balance that with your numeric fitness. That's called model transparency. What you have ended up with is effectively a linear mixture model in the log domain ($\exp(a*\log(x)+b*\log(y))$), which is not uncommon.

If you want to experiment with more general forms for your scoring, also consider using a decision tree or neural network.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.