0
$\begingroup$

I need some advice on the simplest/best way to structure an ML model for a (slightly) non-standard situation.

Setup: I have many teams in a company that have leaders. Each team has two options for a leader (A & B). The teams and their leader options differ a lot across the company, but the choice of A or B for each team is more or less arbitrary/random. I can measure teams' performance (y) consistently and potential leaders' characteristics (x_A, x_B).

Goal: I want to predict what kinds of leaders are best. Should A or B have been chosen?

Analysis: there are a few ways to set this problem up and here's where I need advice. Say f is some ML model function of characteristics, should I fit

  • y ~ f(x_Winner) -- this is primarily the relationship I want, but the model doesn't see what the alternative leader was for each team (just the winner)
  • y ~ f(x_i, T_i) -- that is, have two observations per team, one for A & B each. Add A and B's characteristics and include T_i which is equal to 1 if the person was chosen as leader. The model still doesn't see the alternative and predictions may be inconsistent across observations (f(x_A,1) should = f(x_B,0))
  • y ~ f(x_A, x_B, T_A) -- now the model sees both alternatives, but the ordering of inputs is very arbitrary; does A or B go in the first slot?

I wanted advice on what's most standard to do here and how to evaluate which model is best. Do I care most about whether the model predicts y? or, since each choice is relative, how the predicted effect (f(A wins) - f(B wins)) is most related to y?

Edits: Since the morality of this came up, this is purely a conceptual question. There's no actual organization or teams. If you prefer, think about a situation where I observe a series of binary choices, characteristics about the choices, which choice was made, and the ultimate outcome. I want to predict the 'best choices.' The choice is always binary between A & B, but the labels don't mean anything other than "first choice" and "second choice."

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ It's not clear why you think that the features of the loser should be relevant for your model - assuming the performance of each team depends on the characteristics of it's leader, why does it matter who was not chosen ? The meaning of the "A" and "B" labels is also not clear, is that just random labeling or are there candidates of "type A" and "type B" ? $\endgroup$
    – J. Delaney
    Commented May 6, 2022 at 18:10
  • $\begingroup$ The losers' characteristics ultimately might not be predictive. However, by telling the model about both options, it might learn what was relevant in the change (how A was different from B) and what was simply inevitable (characteristics A & B share). A & B are arbitrary labels not types, each one (in this example) is an individual. $\endgroup$
    – Cory Smith
    Commented May 8, 2022 at 4:15
  • $\begingroup$ Let me know if that's helpful. Part of my question, per above, would simply be the most appropriate metric for assessing model accuracy. When testing about including the loser's characteristics, should I assess accuracy based on the final prediction? or should I assess based on what the model says about the difference between choosing A or B (since the relative choice is all that matters)? $\endgroup$
    – Cory Smith
    Commented May 8, 2022 at 4:29
  • $\begingroup$ The point is that you have no way of knowing if a given choice is correct or not because you don't know what the teams' performance would have been had the other candidate been chosen. The only relevant metric I can see here is the accuracy of the prediction given the characteristics of the leader $\endgroup$
    – J. Delaney
    Commented May 8, 2022 at 11:23
  • $\begingroup$ Your vote for this metric & feature set is helpful & if you put it in an answer with additional logic I will accept it. Thank you. In particular, if you would say more about why you prefer the absolute metric to the relative one I proposed, that is getting at the question. I don't think it's right to say that we can't have any handle on the alternative choices and this means ignoring features. Losers' features may (or may not) have contextual value. Even a simple model y = x+1 makes a prediction for x=2.3 even if we don't see x=2.3 itself. $\endgroup$
    – Cory Smith
    Commented May 8, 2022 at 14:45

1 Answer 1

Reset to default
3
$\begingroup$

It's not a machine learning problem and it is a bad idea.

First, it is ethically dubious to have black-box software to make career decisions that would potentially influence the future of those employees. There are tons of research and real-life examples showing how machine learning algorithms can make biased decisions. Weapons of Math Destruction by Cathy O'Neil is a nice start to learning more.

Second, the choice needs to be based on past performance data. So if you have relevant data on the performance of candidates A and B, you just need to pick the one that performed better. If you don't have the data, you cannot make such a decision. So you need to come up with good metrics (relevant, measurable, etc) and make decisions based on them. You also need to give both candidates an equal chance to prove their worth. If you don't have such data, you don't have grounds for making the decision. If you have the data, you don't need machine learning. If you instead just threw some random data to the algorithm, you are risking getting a biased, useless, garbage in, garbage out result. If you had such data, you could use some simple, explainable algorithm like linear regression that could help you to find out how much should you weigh each of the criteria used for making the decision, but anything above that is dubious.

Finally, if you had such an algorithm, it may be bad for your organization in the long run. As noticed above, you risk legal and reputational costs if the algorithm ends up biased. You also risk that your employees get demotivated if they knew that career decisions are made by an "unpredictable" black-box algorithm. In such a case, people would not know what is expected of them to succeed, what is demotivating. They will not know how the decision is made, but this wouldn't stop them from gaming the system and doing all the crazy things that they perceive that the algorithm considers as advantageous for them.

I agree with you that arbitrary career decisions are not a good thing, but "AI" is not a solution here. I guess that the decisions are arbitrary because you don't have relevant data to make the decision. Whatever machine learning algorithm you pick, it would use the same crappy data available to people who made the decisions in the past. It would also need to learn from the data of "arbitrary" career decisions made by humans and it will end up repeating those decisions because this would be the only reality that it would know from the data. You rather need an organizational change, not machine learning.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This is a really negative and unhelpful response. This is a conceptual question for my learning about structuring predictive models. I don't work in anything related to hiring and firing decisions, I just gave a simple example to illustrate a type of problem. I appreciate that your points are important for broad questions about ML, but it would have been helpful if you'd included these as a sidenote and addressed my main (conceptual) question. Just listing "linear regression" is not an answer to the question I asked which was about structure and metrics, not fit functions. $\endgroup$
    – Cory Smith
    Commented May 8, 2022 at 4:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.