In the interest of transparency I'll answer based on how my team currently does it, without commenting on whether I think this is the "best" way so that it more closely describes real world practices rather than normative ideals.
- 10-fold cross validation grid search, selecting on AUC
- Final models are trained with final parameters and validated on yet another, "true," holdout test set (not used in cross validation)
- No ML cloud providers, but we do use compute-optimized EC2 instances from AWS
- Yes, AUC (mean of AUC from 10 cross-validation or bootstrap passes) for binary classification; otherwise we decide at the start of a project, usually MSE but it depends
- Yes, proprietary imputation for known missing data, 3rd party demographics (income, employment, etc), various feature engineering
- Use caret for R, sklearn for Python. Always start with naive logistic regression to get a baseline to compare to more advanced models. RandomForest can't overfit, so run one over night to get feature importance before proceeding to hyper parameter optimization and model selection: if RF can't use a feature, it probably has no mutual information with the DV. After selecting a reasonably strong model, 90% of improvements come from data cleaning and thoughtful feature engineering. Always calibrate the output of a black box model using a data set with a real world class prevalence.