I'm interested in ML research and want to get some insights into how its done in practice.

In particular:

  1. What kind of data are you dealing with?

  2. How are you dealing with hyper-param optimization?

  3. How are you making sure the work is reproducible?

  4. Are you using a cloud provider to run jobs on the cloud? If so, how are analyzing the results?

  5. Do you have a single number evaluation metric?

  6. Does your work involve building on top of existing models? If so, how do you obtain them

  7. Any other extra tips & tricks, best practices etc.

Thanks for all the helpers!

  • $\begingroup$ This isn't really on-topic here, and the haters are going to rain fire for it. They will say go ask software (stack overflow) about software packages. There is a computational science group you could also ask. $\endgroup$ – EngrStudent - Reinstate Monica Sep 13 '18 at 20:10
  • $\begingroup$ @EngStudent: "Haters"!? Anyway, the problem with this question is that it's a poll with as many "correct" answers as we have users - it's not going to fly on any Stack Exchange site. It'd be fine in chat of course, & probably somewhere on Quora or Reddit. $\endgroup$ – Scortchi - Reinstate Monica Sep 13 '18 at 20:25
  • $\begingroup$ @Matantsuberi: Welcome to Cross Validated! Sorry to have to close your first question - please have a look at our help pages to get an idea what sort of questions the site is for. $\endgroup$ – Scortchi - Reinstate Monica Sep 13 '18 at 20:36

In the interest of transparency I'll answer based on how my team currently does it, without commenting on whether I think this is the "best" way so that it more closely describes real world practices rather than normative ideals.

  1. 10-fold cross validation grid search, selecting on AUC
  2. Final models are trained with final parameters and validated on yet another, "true," holdout test set (not used in cross validation)
  3. No ML cloud providers, but we do use compute-optimized EC2 instances from AWS
  4. Yes, AUC (mean of AUC from 10 cross-validation or bootstrap passes) for binary classification; otherwise we decide at the start of a project, usually MSE but it depends
  5. Yes, proprietary imputation for known missing data, 3rd party demographics (income, employment, etc), various feature engineering
  6. Use caret for R, sklearn for Python. Always start with naive logistic regression to get a baseline to compare to more advanced models. RandomForest can't overfit, so run one over night to get feature importance before proceeding to hyper parameter optimization and model selection: if RF can't use a feature, it probably has no mutual information with the DV. After selecting a reasonably strong model, 90% of improvements come from data cleaning and thoughtful feature engineering. Always calibrate the output of a black box model using a data set with a real world class prevalence.
  • $\begingroup$ RFs can overfit. $\endgroup$ – Laksan Nathan Sep 13 '18 at 18:48
  • $\begingroup$ Can you maybe give some idea as to what kind of data are you dealing with? $\endgroup$ – matan tsuberi Sep 13 '18 at 18:52
  • $\begingroup$ Also, why grid search instead of random search? $\endgroup$ – matan tsuberi Sep 13 '18 at 18:53
  • $\begingroup$ @Inathan. Of course. But random forest does not continue to overfit more and more simply because you increase the number of trees in the forest. Therefore a very lazy way to build a pretty good model is to crank the number of trees way up and run over night. The resulting RF model can be used for certain things, such as estimating feature importance and getting a reasonable estimate for how good a final model. Such a bloated RF model will not be production quality but is an excellent low-effort first step. $\endgroup$ – olooney Sep 13 '18 at 19:11
  • $\begingroup$ @matantsuberi CRM data. We use machine learning mostly to predict propensity to convert, purchase, or utilize various services for a populations of current and prospective customers, estimate lifetime value, predict churn and retention, etc. $\endgroup$ – olooney Sep 13 '18 at 19:19

Not the answer you're looking for? Browse other questions tagged or ask your own question.