What are some good questions to ask when assessing an ML model? Let's say a colleague tells me they has a good ML model and show me some numbers that suggests that they are right. What are some good questions that I should ask given that I can't LOOK at the actual model or data. Basically I would want to understand that the model is not overfitted to historical data, that they´ve been cognizant of selection bias etc, and what steps they have taken to prevent that.
I think I have a fairly good idea, but I would like to have a second opinion.
 A: A "good" ML model is context specific (e.g., high ROC score is not sufficient to declare a model is a great classifier for your particular use case). Some of this you can ask the person who developed the model, but others are things you need to ask before using the model.
Now, since they have already tested it on a test set, you can gauge a couple things:


*

*What was their evaluation metric? (i.e., the metric: RMS, ROC, MAE,
MAPE, F1, precision, recall, etc)

*Why did they choose this metric?

*Is this metric appropriate for your application?

*Where did they get their data?

*Did the data contain duplicates or near-duplicates? Where they
removed? What were the most important predictors in the model?

*What was the degree of missingness for each predictor? How was
missing data handled?

*How representative is the data compared to the population it will be
applied to (sometimes data is based on a convenience sample, and
hence very biased) How did they correct any biases in the dataset to
ensure it is trained on a relevant population?

*Is performance stratified in some way (did they even try to check?)
or is it generally uniform across most subsets of the feature space?

*How did they form the training and test splits?  Why did/didn't they
consider cross-validation or bootstrap validation (bootstrap
replications of train/test splits)

*Were there substantial differences between the test and train data
 distributions? Test vs train accuracy (to gauge overfitting)

*If they did hyperparameter tuning, are the results you are seeing
 from a held-out validation set (i.e., data not used as part of
 train or tuning phases)?

*Were there possibilities for information leakage between train and
 test sets or within a training set (e.g.,, did they create features
 that incorporate target information, not just predictor
 information)?


And possibly many more depending on how you plan to use the model (e.g., run time complexity, memory complexity, distributed or single core, uses data you can't get, etc)
A: Reading "historical" I assume that you are talking about time series. In this case it's important that out of sample test subset is never located earlier in time than training subset. 
Dataset size is important and should be in proper relation with model's number of parameters.
I'd also ask if chosen features include something "improper", like stock's absolute price or person's social insurance number (just an example).
