What general procedures are out there for quantifying how well an estimator (such as for the mean, standard deviation or correlation) of a continuous random variable gives a consistent picture of its respective true parameter out-of-sample as it did in-sample?
I understand it's all down to how the new unseen data is different than it was at the time of estimation, but generally it can become commonplace for people to identify "metric 1, the sample mean, is much more difficult to estimate than metric 2, the sample variance" for example. Are those conclusions more often arrived at through theoretical introspection rather than empirical realization? given that, if you measure something, you'd think we would just accept it as is since we carried out the calculation and that's what we got.
Rather than procedures, actually, (like cross-validation, Monte Carlo) are there instead performance measures that directly provide inference about the reliability of an estimator?