1
$\begingroup$

Am working on a simple logistic regression with 1000 records and 28 features.

My business users suggest that they want to first see what the AI can do by itself based on our data as it is. Meaning, they don't want me to do feature engineering, trying out multiple algorithms etc.

They want me to avoid all that because they feel it takes time to do feature engineering and they wish to showcase something quicker and earlier. For the 1st cut, they wish to go live with baseline model with no feature engineering (even if it is 50% accuracy).

They are okay with 50% accuracy because currently there is nothing done to solve this problem. No one is tackling this problem or even thought to solve this problem. So, this is new to them... So, even if it is 50%, they feel it is something good for them to start. Meaning, they identify those 50 cases and go follow up with them. Since this 50% is reliable (for them), meaning certain actual positives and actual negatives predicted correctly, they wish to go live with this.(and focus on that 50% cases)

Of course, they suggest me do feature engineering, model experimentation etc after going live. By live, I mean just a simple static dashboard (and not high end MLops etc)

Is this a right way to go further? As a novice data scientist, I don't feel right about this. If it had been at least 80% (my random choice), I would have been bit okay. I don't have any evidence to proof that 80% is the right choice rather than just saying higher no of actuals are predicted correctly.

So my questions are

a) what should I do and what are the pitfalls/points that I should make sure to keep them aware?

b) Is there anything important that I should highlight them?

c) Should this project still be dropped if business is okay to be with 50% acc? Can we continue to use this model as long as business is fine with it?

d) Any real time experience from your model deployment decisions?

Can share your views on this? Would really be helpful for me to learn and also keep them aware?

$\endgroup$
6
  • $\begingroup$ If the business really is OK with a 50% accurate model, why not just tell them to spare you the time and production costs and just flip coins, or just predict the majority class? It would be more helpful to know what purpose the model is serving prior to giving advice. $\endgroup$ Commented Jan 28, 2022 at 4:43
  • $\begingroup$ Based on your experience, when you deploy models, what do you consider, take into account? Is there any way to quantify this 50% accuracy? Meaning, instance id = 2 (actual positive), is correctly predicted as positive with 95% CONFIDENCE? Is there anyway to get this confidence % for each instance? $\endgroup$
    – The Great
    Commented Jan 28, 2022 at 4:49
  • $\begingroup$ let's consider the accuracy is 80% (and the class is imbalanced)..recall for positive class is 87% and recall for negative class is just 20%..Business can say that will very well trust model for positive cases...because it has predicted most of the positives correctly...This gave them some confidence in the model. So, now for the recall of negative classes, even though it is 20%, they think they can trust this 20% (despite low representation of negative class) because when it has done well for positive class given enough representation, it would do the same for negative classes. $\endgroup$
    – The Great
    Commented Jan 28, 2022 at 4:55
  • $\begingroup$ So, now let's say their business objective is to focus on positives now, this model does serve its purpose at a recall of 80%. So, they are happy about it. Again, we will do al feature engineering and experimentation, algo improvement in upcoming stages but they just wish to go ahead with this for now.. $\endgroup$
    – The Great
    Commented Jan 28, 2022 at 4:58
  • $\begingroup$ @DemetriPananos - they say random coin flip doesn't give you the ranks/likelihood of an event happening. or it takes a lot of time to compute the likelihood.. Meaning, if I run a logistic regression, for 50% of the cases correctly identified, I will also get a likelihood/probability, that helps to them to rank those cases (and follow up with high risk once first). If it's random coin flip, they might be focusing on the low risk ones (with no return) etc. $\endgroup$
    – The Great
    Commented Jan 28, 2022 at 5:06

1 Answer 1

5
$\begingroup$

a) what should I do and what are the pitfalls/points that I should make sure to keep them aware?

In my opinion, you should do what they want.

b) Is there anything important that I should highlight them?

It's a bad idea to focus on accuracy. From what you said in the comments, it seems that they want a measure of how well the model ranks cases, in which case you should use AUC or a similar measure. Accuracy is usually misleading anyway.

c) Should this project still be dropped if business is okay to be with 50% acc? Can we continue to use this model as long as business is fine with it?

Why not? It's not your decision. It's up to the business.

d) Any real time experience from your model deployment decisions?

It sounds like this business might have been burned by data scientists before. They hire someone who treats the task like a Kaggle competition and takes weeks to build the best model they can. Or, they hire a statistician, who spends the entire time whining that the data isn't good enough (following Fisher's maxim about "what the experiment died of") and doesn't do anything at all.

Most businesses don't really want that. They want to get a minimal working prototype into production as quickly as possible. Then it can be improved later. Or maybe the situation is hopeless, in which case they can make the decision to give up quickly.

I worked with one guy who spent his time building an extremely complicated Hidden Markov Model. That's what he did. I worked with him for five years. In that time, as far as I am aware, he didn't put a single model into production. He just came to the office every day and added nodes to his model.

Now he's in another company, managing data science teams. Please don't be like him.

$\endgroup$
1
  • $\begingroup$ Wow..such good and direct response to each of my questions. very useful to know. Upvoted $\endgroup$
    – The Great
    Commented Jan 28, 2022 at 7:34

Not the answer you're looking for? Browse other questions tagged or ask your own question.