1
$\begingroup$

I am trying to find an individual agent's score based on customer scores (raters) while the customer only rates their overall experience.

Context

The customer enters a shop and meets a service agent number one. Then he moves onto service agent two and so on. Generally, a customer meets 4 to 6 agents one after another and once he is done with all the transaction he is asked his overall experience of all the agent interactions on a scale of 1 to 5. It is possible to know which all service agents the customer met and tie the overall score to all those service agents

It is optional for customers to leave feedback. Moreover, there are multiple service agents, with different shift routines, close to 200, that are working in different departments which implies that each customer is unique but service agents are randomly assigned.
Eg - Customer no. 1 meets service agent A,B,C,D and E, while customer no. 2 meets B,D,E,G and H.

Problem

  1. I am trying to ascertain the rating of each service agent based on the overall customer store
  2. Additionally, it would be great if I could somehow compare service agents across departments

Attempt

I have been assuming that each service agent is equally liable for the overall experience score and thus I have been doing the following

  1. Assign the overall experience score to each service agent who met that customer
  2. Find the average of all such customer scores for that service agent

Customer Score Table
Sample Table

Service Agent Score Table
Service Agent ║ Average Score
║ A ║ 3.67
║ B ║ 4
║ C ║ 3
║ D ║ 5
║ E ║ 2.5

I have also searched stack exchange and come across terms such as Cronbach's alpha, Cohen's Kappa and Fleiss Kappa but I do not think they apply to my case. I think this way of rating plus ANOVA can do the trick as can be found here which is somewhat similar to my problem.

Your thoughts?

Improve this question
$\endgroup$
4
  • $\begingroup$ Seems to me this is not going to work well. A perceived very bad experience with only one agent can spoil a customer's ratings for the whole visit---even if the one agent's behavior is not typical of the group. // Personal anecdote: Dealership where I get my car serviced has one persistently disagreeable agent and several pleasant, efficient ones. Always asked to send in questionnaire. Don't give 'very highly satisfied' rating for whole visit even if outcome satisfactory: that one agent is enough to make me want to stay away. Mgmt must be clueless not to realize they've got a grump. $\endgroup$
    – BruceET
    Commented Oct 4, 2020 at 21:12
  • $\begingroup$ @BruceET We had thought about it but couldn't because of teo reasons. First, multiple questions leads to frustration among the responders which will cause lesser response rate. Second, most customers aren't bothered to know the names of all the service agents who they met even if they arr willing to fill a longer survey. $\endgroup$
    – Ctrl
    Commented Oct 5, 2020 at 4:14
  • $\begingroup$ I wasn't criticizing your survey, just pointing out difficulties of making inferences about specific employees. Even so, it you find a particular store with an unusual mix of good and bad scores, then it may be worthwhile trying to deduce who was on duty for which questionnaires. $\endgroup$
    – BruceET
    Commented Oct 5, 2020 at 7:45
  • $\begingroup$ @BruceET Deducing who was on duty is something that is already being monitored. The difficult part is correlating it with the the customer score. Are you aware of any such problem or a way to solve this one with the constraints mentioned? $\endgroup$
    – Ctrl
    Commented Oct 5, 2020 at 8:57

1 Answer 1

Reset to default
1
$\begingroup$

I wasn't able to find a satisfactory answer but I suppose statistics like Shapley Values or Wins above replacement levels might be helpful.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.