2
$\begingroup$

There are an infinitude of tutorials on how to use this and that statistic, but I can’t find anything at all about how to invent a statistic, and validate it. I can guess, from the papers: You make up a scoring algorithm, and then use simulation to characterize it, and unsure that it meets my, and some set of required desiderata. I can a guess what those sims entail, but usually the code for those isn’t provided. Is there a textbook, or even a clear explanation of exactly how to go about this? It looks like the folks who do it know something that I can find nowhere clearly explained. Is there an advanced course called “Inventing new statistics”, or does everyone who does it just glean how it’s done by reading papers of others who have done it? Nb., I’m talking about statistics here, not new ML methods. I know how to run a cross-validation. What I don’t know, and can’t find clearly explained, is how to figure out how to characterize and validate a statistic.

To be more specific, given that there are an infinitude of functions, and probably a somewhat smaller infinitude that might do what I want, how do I decide between different members of the smaller group, and then prove that it performs the basic desiderata of a general statistic. Let me be more specific: Suppose that we didn’t have a t-test, but wanted to invent the t statistic, a function that generates a score that can be interpreted as a p-value - i mean, today, armed with … Python or R (but, obviously no built in t-testing packages) … not how it was done historically. What steps would one take to do this?

$\endgroup$
2
  • $\begingroup$ I've provided an answer below. If you mean something more specific than a general statistic (i.e. any function of data), then I suggest editing your question to be more specific about what kind of statistic you are interested in. $\endgroup$
    – jkpate
    Aug 17, 2021 at 9:55
  • $\begingroup$ @jackisquizzical Are you in the situation of having some distributional model or not? $\endgroup$
    – Glen_b
    Aug 17, 2021 at 13:33

2 Answers 2

2
$\begingroup$

This is a great question. I would say it starts with a deep study of existing statistics for other problems, including hypothesis tests, point estimates, and confidence intervals. For a given task, what makes one statistic preferable to another? In hypothesis testing we are interested in a most powerful test that maintains type I error control (the p-value is uniformly distributed under the null). For point estimation we often seek an estimator that is unbiased or perhaps median unbiased and has the smallest variance in this class. Confidence intervals can be thought of as the inversion of a hypothesis test, so we are interested in the shortest intervals that maintain their nominal coverage level.

For a very new and challenging problem one can simply propose a new statistic that meets only some of these criteria. For instance, a hypothesis test where the p-value is uniformly distributed only asymptotically and power is acceptable. Later work will show the merits of the statistic relative to other proposals. Sometimes a method that is not exact or unbiased or most powerful is still preferable because of its ease of use and interpretation.

By reading and understanding other's work you will develop a preference in approach and naturally see opportunities to improve their methods.

$\endgroup$
2
$\begingroup$

A statistic is just a function of data. To invent a new statistic, all you need to do is define the function. The procedure for understanding your statistic will depend on what you want to understand. It could involve a theoretical analysis (proving mathematically that it is a member of some class of functions, for example), analyzing the behavior of your statistic on simulated data, and/or analyzing the behavior of your statistic on real data.

$\endgroup$
4
  • $\begingroup$ Thank you. I think I understood it this far. As you suggest I’ve clarified the question. $\endgroup$ Aug 17, 2021 at 15:53
  • $\begingroup$ Unfortunately, it is almost certain that a statistic invented in this way will lead to inadmissible procedures. That's why successful statisticians first study the underlying statistical theory and apply it to developing statistics. $\endgroup$
    – whuber
    Aug 18, 2021 at 12:22
  • $\begingroup$ @whuber: See, that's what I'm asking after. Seems like there should be a course or text specifically targeting this sort of thing. Like, there are innumerable programming texts and online pointers on how to engineer programs and avoid problems. Seems like statistic invention is a kind of engineering; why aren't there dozens, or even hundreds of books and blogs on it? It's frustrating to us engineers; we're only allowed to use the (metaphorically) packages (i.e., statistics) that someone else built, but no one will tell us clearly how to build our own! $\endgroup$ Aug 18, 2021 at 16:41
  • 1
    $\begingroup$ Find the website for a top-rated PhD-granting statistics department and look at what it offers on statistical inference and estimation to entering students: that will give you an indication of what people who develop and research statistics need to know. $\endgroup$
    – whuber
    Aug 18, 2021 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.