Fair enough. I guess I was hoping for an ex cathedra answer along the lines of "that is impossible" or "that is provably true", but I'll also settle for empirical evidence. Thanks for the feedback!

@Sycorax But that seems to be contrary to the quite clear statement that I've seen before: that regularization only has a (positive) effect on validation performance, and can only /decrease/ training performance.

@amoeba I see, thank you very much for clearing this up. The patience of contributors on this site with imbeciles like me and the questions they ask is admirable.

Thanks. I skimmed the van der Maaten/Hinton one before but didn't notice they cover Isomap. One question though: I want to visualize/interpret the global structure as much as that is possible. So t-SNE performing better at clustering the different classes (and avoiding the crowding problem) is definitely useful, but I shouldn't draw too many conclusions from the organization of the clusters in relation to each other, right? Put differently: if the goal is less aiming for (optimal) local clustering, and more about interpretable differences on a larger scale, is t-SNE still the right choice?

Second, a proof of deep network's having an 'advantage' when approximating hierarchically computed functions; see: Mhaskar et al, "Learning Real and Boolean Functions: When Is Deep Better Than Shallow", 2016. Now, I realize, these two results don't provide a 'hard' measure like crossvalidation techniques do, but I thought maybe it's still plausible to say they are related to the generalization question: answering formally (though perhaps based on some "impractical" assumptions for actual ML) how network depth and model performance on some class of functions are related. (4/4)

Maybe it's time to give an example of what I (vaguely) had in mind until now: there's an active branch of research that could perhaps be summarized as "formal justification of ML model performance". Two examples (both related to neural networks, sorry for that): First, the "space folding" results (that's the very basic intuition) by Montufar and Pascanu, using, roughly: combinatorics + some topology; see: Montufar et al, "On the number of linear regions of deep neural networks", 2014. (3/4)

2. You write: "Unfortunately there is no simple rule for knowing how close your training data will represent production data. You just have to use good judgement". This is related to the need to balance the goodness of fit of a model with model simplicity, correct? Working from the assumption that more complex models are more likely to overfit. My question then is: aside from such 'balancing' grounded entirely on intuition, are there any formal (or semi-formal) approaches that people here can think of? (2/4)

Thanks for your answer to my question. Two follow-ups: 1. You mention holdout sets, which fall within the context of cross-validation (mentioned by Kodiologist as well). That seems to be the most common technique to evaluate the 'generalization' of a model, but what I wanted to know: is there any other standard technique that does not fall under the crossvalidation umbrella, that I simply don't know about? Maybe to ML veterans this knowledge, that this is the only technique, is so self-evident it doesn't bear mentioning, but I'm still pretty ignorant on these things. (1/4)

Thank you for answering. (I'd upvote it, but cannot yet on here). I know (or at least: heard of) the techniques you mention, but I guess I was thinking of a slightly broader view on the question. Then again, as you pointed out above in a comment to my question, that might be more philosophy of statistics than statistics proper. I'll leave the question open for another few days, but will mark your answer as accepted o/w.

@Kodiologist Thank you for making it through my wall of text. I should maybe say, I am not necessarily asking about the philosophical side of it alone: putting it more succinctly, I wonder which other ways to evaluate models are used, other than the train/test data separation mentioned in my last bullet point, that can be seen as related to the question of generalization?