1
$\begingroup$

I have following environment that I'd like to analyze.

My company is divided in two teams:

  • Insert Team (composed by 150 persons)
  • Quality Check Team (composed by 10 persons)

Suppose that all people of first team compile a form with 10 text boxes, for example: name, surname, address, category, studies, preferences, hobbies, etc., and the second team (quality) analyzes a variable percentage of the data which are inserted.

Now I'd like to know which method could I apply to find good or poor members of first team, as analyzed by second team, captured using historical feedback of all of the analysis made before.

An example to understand better:

If the first team inserts 10 data points for the users, then we'll have 1500 data points to analyze. Suppose we define 10% of those data to pass to the second team. After their analysis and feedback, I'd like to use it to influence the next 10% of data capturing, for example, to focus attention on users that are less precise or on text box where there are more errors (which is the approach of major companies like Facebook or Twitter).

|cite|improve this question
$\endgroup$
  • $\begingroup$ I don't think this question is necesarily about statistics--and even less about stat. sig., or history. It seems to be about measurement and/or evaluation. You're looking for a good method by which to measure performance on a task, right?. Unfortunately you can hardly use statistics to determine what sort of responses are "precise"--that's going to require collaborative thought, hopefully leading to a systematic approach to evaluating the forms. $\endgroup$ – rolando2 Dec 11 '12 at 1:28
  • $\begingroup$ It's not a single task. It's multiple tasks on n forms. I have made this kind of example to make it easy. Do you think it's not a right place where post a question? $\endgroup$ – Luigi Saggese Dec 11 '12 at 9:04
  • $\begingroup$ Can the Quality Check team make a definitive judgement on each text box's accuracy? And will they be 'correct/incorrect' judgements or something with more distinction? $\endgroup$ – Peter Ellis Dec 11 '12 at 9:15
  • 1
    $\begingroup$ The weighting seems a bit weird; why not simply use all the data points at once to find an overall error rate per member of team 1? Of course, this error rate could be updated for every member of the first team when they add another 10 data points. $\endgroup$ – PascalVKooten Dec 16 '12 at 14:06
  • 1
    $\begingroup$ This answer's going to be lame because I don't have time to go into detail, but I think you should look up "active learning" and "inter-annotator agreement". Good luck. $\endgroup$ – Josh Hansen Dec 17 '12 at 18:53
2
+25
$\begingroup$

I'm going to assume the quality control team just make a correct/incorrect judgement about each text box. If so, this looks like a good candidate for a straightforward logistic regression model, with text box as one explanatory variable with ten levels, and team member as a second explanatory variable with 150 levels. The response will be one if the entry is wrong, zero if it is right.

The estimates for the coefficient for each of the various explanatory variables can then be used to identify which team members and which text boxes have higher error rates. This should then be fed into an algorithm for determining weighting for the future sampling. Weighted sampling is a straightforward technique. You will need some limits on the weighting - you don't want any text box to get below say 1/20 chance of being sampled, or any individual team member to get below 1/500 - so you choose an algorithm that sets those (or some other arbitrary levels) as the minimum for the weighting.

Addition

If the quality control team make a more nuanced judgement - say a rating on a four point scale from "poor" to "perfect" - then the procedure would be just the same as above but you could use ordinal logistic regression instead. The response variable would be this rating; the explanatory variables would still be the factors for team members and for text boxes. You would still then need to use the estimated sizes of the effects for each level of the factor in an arbitrary algorithm that weights the chance of those text boxes and those users being included in the next sample.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ If i'd like to have not a dichotomous judgement on each text box (like 1 or 0) but multiple, how could i change this method? $\endgroup$ – Luigi Saggese Dec 12 '12 at 11:52
  • 1
    $\begingroup$ Instead of logistic regression, just use the appropriate form of regression for whatever the response is. For example, ordinal logistic regression. $\endgroup$ – Peter Ellis Dec 12 '12 at 19:34
  • $\begingroup$ Could you explain me ordinal logistic regression model in my use case :-) $\endgroup$ – Luigi Saggese Dec 13 '12 at 10:46
  • $\begingroup$ Hi Luigi - I've added a paragraph on this but really there's not much to say. $\endgroup$ – Peter Ellis Dec 14 '12 at 21:31
  • $\begingroup$ I don't understand the difference... $\endgroup$ – Luigi Saggese Dec 17 '12 at 8:58
1
$\begingroup$

Phase 1: Rank the likelihood of errors for each input team member

Use poisson regression where each person in the first team is a binary predictor variable and while the number of incorrect forms/fields is the response variable. However, you will need to have significant historical data in order to make this inference. Ordering the team members by the regression coefficients (descending) is going to give you the ranking.

Phase 2: Use the ranking for sampling for quality team

Once you have identified the regression coefficients (r1, ..., rm), you can create a probability vector (p1, ..., pm) := ((r1, ..., rm) + min(r1, ..., rm)) / (sum(r1, ..., rm) + m*min(r1, ..., rm)). Then, if you the quality team can handle #n validations, you should distribute them according to multinomial distribution with MN(n, (p1, ..., pm)).

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.