# Normalizing a data set by using z-scores from a related data set

I am trying to come up with a normalization for a set of data. My company puts our products through a series of testers (people) to score our products on key metrics. I am able to access all records from each tester and when I look at the averages, I find that some testers always have scores higher than the standard deviation and some testers always have scores below the standard deviation. Since we only have 3-4 testers on each product, this variance in testers could impact our decision on products.

I was able to calculate a Z-Score for each tester. I was thinking about using this to create some form of normalization factor that I could apply to their raw scores. I didn't think using a z-score transformation was the right approach as this process is using a z-score from a different data set. I want the normalization to be using the testers scores and applying that to the products overall score.

Is this the right approach? I'm not the best at statistics so maybe there is a totally different method that works better.

Thank you

Further info: Testers will have already tested hundreds of products (depends on how long they've been on staff). Products are randomly assigned to testers, testers never test the same product twice. They will typically score on 8 keys areas on a scale of 1-5 per product. We take an average of these scores to come up with the tester's final score (average). The products are then then sent to other testers for their averages. The testers scores are all averaged together to get a final score.

I know sometimes averaging averages can be bad, but in this case, the scale of ratings (1-5) never changes, so I believe it works here.

• Citanaf, I am so sorry that I put my comment as an answer to your thoroughly formulated question. Your idea sounds immediately good to me - you are calculating the z score individually, for each tester, right? :-)
– Ute
Commented Jul 25, 2023 at 7:15
• A zscore is being calculated individually for each tester, correct. The question is how to apply that zscore to their ratings I think. Commented Jul 25, 2023 at 14:29
• Citanaf, I think your approach does make sense (unless your testers always give the same ratings). I will have a look how it behaves in a simulation. / / Have you considered the tag rating for your question? It might be relevant.
– Ute
Commented Jul 25, 2023 at 17:08
• Is there a possibility some testers might be better than others? Would you have any objective data about that?
– whuber
Commented Jul 25, 2023 at 17:58

I interpret your approach as fitting a simple model to existing data. Subsequently you apply the fitted model to predict parameters for new products, which is perfectly OK.

This is the model, in technical terms:

• Each product has a hidden "true" quality measure $$x$$ (the z-score).
• For a randomly chosen product from the (hypothetical) population, containing the known and all possible future products, we get a random quality measure $$X$$. This random variable has mean $$\mathrm{E}X = 0$$ and variance $$\mathrm{Var} X=1$$, but is not necessarily normal distributed.
• When the product $$i$$ is tested by tester $$j$$ , it gets assigned and average score $$Y_{ij}$$ within $$[0, 5]$$. This average score depends on the tester - some tend to give higher scores, some tend to give very similar scores to almost all products. And then there are random fluctuations on top of it. Let $$\mu_j$$ be the (theoretical) mean score given by tester $$j$$, and $$\sigma_j$$ be a scale factor, and $$E_{ij}$$ the random fluctuations, then $$Y_{ij} = \sigma_j x_i + \mu_j + E_{ij}.$$ Your proceed now by estimating $$\mu_j$$ and $$\sigma_j$$ for each tester from the data, and for a new product, you predict the score $$x$$ from observed $$y_{ij}$$. I don't know exactly how you are doing this, but probably you have noticed that you get different results depending on $$j$$ when you just invert the model and calculate $$\hat{x}_{i(j)} = \frac{y_{ij} - \hat\mu_j}{\hat\sigma_j}.$$ As a quick fix for discrepancies, I suggest to to just take the average of these fitted values as your guess for the true z-score, $$\hat{x_i} = \frac{1}{n_i}\sum_{j=1}^{n_i}\frac{y_{ij} - \hat\mu_j}{\hat\sigma_j}.$$ Here $$j=1,\dots, n_i$$ indexes the testers that scored product $$i$$.

Your problem goes under the name inverse estimation or inverse prediction. Only few threads here on cv deal with this topic, possibly because methods for inverse estimation are less commonly used (and taught) than ordinary regression problems. Typically it concernes normal distributed data with no replicates for $$i$$ and no experimental design as yours, also in this post with an answer by @kjetilbhalvorsen. You can find an introduction into the topic in Greenwell and Kabban (2014), "investr: An R Package for Inverse Estimation" (The R Journal Vol. 6/1)

Sorry, I accidentially posted a comment as answer, originally (which kind of forced me to write an answer :-). Here is the first version (needed to understand the first comments to this answer): "Welcome to cv, Citanaf! Can you give us a rough idea about the dimension of the data you have? How many products have testers already tested, typically?"
• Hello Ute, thank you for your reply. Testers will have already tested hundreds of products (depends on how long they've been on staff). Products are randomly assigned to testers, testers never test the same product twice. They will typically score on 8 keys areas on a scale of 1-5 per product. We take an average of these scores to come up with the tester's final score (average). The products are then then sent to other testers for their averages. The testers scores are all averaged together to get a final score. I will also add this to OP Commented Jul 25, 2023 at 0:34
• "I will also add this to OP" - Great! so the final score is an average from 8 - all on a likert scale?
– Ute
Commented Jul 25, 2023 at 0:40
• Sorry if I was a bit unclear. There are 8 things that the tester scores on IE: "Ease of Use", "Price", "Build Quality" etc.. these are scored from 1-5. We then average each of these scores for a tester's overall score which will be from 1-5 Commented Jul 25, 2023 at 0:45
• Hi @Citanaf, here comes a real answer - hope it helps. Sorry I accidentially posted a comment as "answer" earlier. Btw, you are still unregistered - hope you can read this anyway :-) You might have missed a confirmation email from StackExchange - could have been caught by a spam filter.
– Ute
Commented Jul 27, 2023 at 21:28
• I just got around to this as I had some things come up. I actually was able to implement this exactly as you recommended. It was quite easy, but I just didn't know how to describe my need in mathematical terms! Thanks so much for your answer! Commented Oct 27, 2023 at 23:49