How can I learn how to create a new statistic? There are an infinitude of tutorials on how to use this and that statistic, but I can’t find anything at all about how to invent a statistic, and validate it. I can guess, from the papers: You make up a scoring algorithm, and then use simulation to characterize it, and unsure that it meets my, and some set of required desiderata. I can a guess what those sims entail, but usually the code for those isn’t provided. Is there a textbook, or even a clear explanation of exactly how to go about this? It looks like the folks who do it know something that I can find nowhere clearly explained. Is there an advanced course called “Inventing new statistics”, or does everyone who does it just glean how it’s done by reading papers of others who have done it? Nb., I’m talking about statistics here, not new ML methods. I know how to run a cross-validation. What I don’t know, and can’t find clearly explained, is how to figure out how to characterize and validate a statistic.
To be more specific, given that there are an infinitude of functions, and probably a somewhat smaller infinitude that might do what I want, how do I decide between different members of the smaller group, and then prove that it performs the basic desiderata of a general statistic. Let me be more specific: Suppose that we didn’t have a t-test, but wanted to invent the t statistic, a function that generates a score that can be interpreted as a p-value - i mean, today, armed with … Python or R (but, obviously no built in t-testing packages) … not how it was done historically. What steps would one take to do this?
 A: A statistic is just a function of data. To invent a new statistic, all you need to do is define the function. The procedure for understanding your statistic will depend on what you want to understand. It could involve a theoretical analysis (proving mathematically that it is a member of some class of functions, for example), analyzing the behavior of your statistic on simulated data, and/or analyzing the behavior of your statistic on real data.
A: This is a great question.  I would say it starts with a deep study of existing statistics for other problems, including hypothesis tests, point estimates, and confidence intervals.  For a given task, what makes one statistic preferable to another?  In hypothesis testing we are interested in a most powerful test that maintains type I error control (the p-value is uniformly distributed under the null).  For point estimation we often seek an estimator that is unbiased or perhaps median unbiased and has the smallest variance in this class.  Confidence intervals can be thought of as the inversion of a hypothesis test, so we are interested in the shortest intervals that maintain their nominal coverage level.
For a very new and challenging problem one can simply propose a new statistic that meets only some of these criteria.  For instance, a hypothesis test where the p-value is uniformly distributed only asymptotically and power is acceptable.  Later work will show the merits of the statistic relative to other proposals.  Sometimes a method that is not exact or unbiased or most powerful is still preferable because of its ease of use and interpretation.
By reading and understanding other's work you will develop a preference in approach and naturally see opportunities to improve their methods.
