I apologize in advance for the possibly philosophical nature of the question, however I would like to get answers from this website rather than from Philosophy exchange at the moment. I also come from a Pure mathematics background, so I might have some misconceptions about how the field of Statistics works. The scientific method and the axiomatic method of mathematics try to address the questions of distinguishing science from pseudoscience, and mathematics from "pseudo-mathematics" respectively. I wish to know an answer for the analogous question of distinguishing Statistics from "Pseudo Statistics". I will present below a potential answer to the previous questions, and I would like to hear from statisticians on this website whether I have the right answer or not.

My idea of the scientific method is basically the flowchart below:

enter image description here

For example, a scientist may be living on two dimensional surface where she always observes that the sum of angles of any triangle add up to 180 degrees. From this observation, our scientist might make the hypothesis that the surface has a euclidean geometry. This hypothesis will make a prediction that for example Pythagoras theorem is true in our surface. The scientist will test the validity of this prediction by making more experiments . If the prediction turns out to be false the hypothesis is thrown away. If the prediction turns out to agree with further experiments, then the scientist's faith in her model increases a bit and she uses the hypothesis to make further prediction and tests them in an ongoing process.

I try to modify the above scenario for statistics.

  1. Step 1: We collect some observations about a real world phenomenon. The phenomenon can be flipping a coin
  2. Step 2: The hypothesis could be that this real world phenomenon of flipping a coin $n$ times can be modelled as Bernouli process $B(n,\frac{1}{2})$. I think of hypothesis making as the process of matching mathematical structures to real world phenomena. In the above scenario, we were concerned with matching the real world phenomenon of the surface we live on to some mathematical Geometric structure, and here we are concerned with matching the real world phenomenon of flipping a coin to some mathematical probabilistic structure.
  3. Next step is to make a prediction. We agree by convention on some significance level before the start of the investigation, which could be say $99$%. If some event $E$ in the probabilistic model of our hypothesis has $P(E)\geq 0.99$, then it counts as a prediction. In our case for example, Chebyshev's inequality will give that $$P(\text{proportion of heads after $10,000$ trials will lie in the interval $]0.45,0.55[$})\geq 0.99$$. Thus, we have our prediction that if we flip the coin $10,000$ times then proportion of heads will be between 0.45,0.55
  4. Step 4: We test our prediction experimentally. We flip the coin actually 10,000 times. If the prediction agrees with empirical results then our faith in our hypothesis increases. If prediction does not agree with empirical results then hypothesis is thrown away and further investigation should take place. Maybe the coin is biased and our probabilistic model should have been $B(n,p)$, maybe the coin tosses affects the later tosses if for example the coin temperature changes or its shape changes thus changing its mechanics, hence we the assumption of independence of coin tosses is inappropriate and does not yield good predictions...etc.

Question : Do I have the correct idea about how statistics works ? If yes, on what basis do we choose the significance level ?

  • 1
    $\begingroup$ I think what you are referring to is covered in en.wikipedia.org/wiki/Foundations_of_statistics, see bayesian inference, frequentist inference (and competing hybrid fisher / neyman schools) $\endgroup$
    – seanv507
    Feb 17, 2023 at 13:26
  • 2
    $\begingroup$ While my answer is a bit critical about the use of a general 'the statistical method', this might be just a semantical discussion and misunderstanding of what you mean. Possibly you mean 'statistics' in some more narrow sense of a particular method or field within statistics. The concept of reverse probability, and methods to deal with it, might be related (What exactly does the term "inverse probability" mean?). $\endgroup$ Feb 17, 2023 at 14:13

1 Answer 1


What is the Statistical Method?

There is no the statistical method. Your title question is asking for something that doesn't exist.

I try to modify the above scenario for statistics.

Your scenario looks much like a Popperian view of the scientific method. It is not a general view of the scientific method (in the recent century several philosophers have been disagreeing with Popper) and it is certainly not statistics. You can involve statistics in it like in your modified scenario but it is not equivalent.

The term statistics originates from the word 'state' and it used to refer to 'data of the state' and evolved into 'the collection of facts'.

Zu Statist ‘Staatsmann’ wird ... bald (Anfang 18. Jh.) in der Form in der Form Statistik f. für ‘Beschreibung eines Staates, eines Landes’ und (2. Hälfte 18. Jh.) die ‘Wissenschaft von der Erfassung, Erforschung und Beschreibung von Massenerscheinungen in Natur und Gesellschaft’ ... bezeichnet

(Quote from the online Etymologisches Wörterbuch des Deutschen)

The terms statistics, relating originally just to data of the state, now relates more generally to the methods that revolve about the sampling, observation, reporting and analyzing the quantitative data in any field.

Statistics is a broad field and can involve mathematical topics such as computing the probabilities of specific types of models of the sampling process and models of observation, but also topics like computer science (making computations, working with databases), social science (developing interview techniques and sampling methods), and visualization (what sort graphs are best suited to convey data).

The scenario that you describe, where the statistician computes a p-value and determines a significance level, seems to relate to a narrow part of statistics which is called 'hypothesis testing'. It is not so much the method of statistics as statistics involves more than just that. And in addition, it is also not belonging solely to statistics. Hypothesis testing is also a part of the philosophy of science. Statisticians relate more to designing the models for performing the computations in hypothesis testing, but it are scientists in a particular field that make a decision to use a hypothesis test. It is not the statistician that says that hypothesis testing is the way to go.

on what basis do we choose the significance level ?

These levels are chosen based on experience and to balance the probabilities between false positive and false negative cases. For instance, in particle physics they now use a $5\sigma$ confidence level and it evolved into that because they perform so many experiments that they would observe too many false positive discoveries if they did not use a strict level (see Origin of "5$\sigma$" threshold for accepting evidence in particle physics?).

  • 1
    $\begingroup$ @NickCox Thomas Kuhn's The structure of scientific theories speaks about a much more diffuse and less linear proces. When we speak about the significance levels this becomes relevant. Those levels may not be applied consistently or systematically, as shown by all sorts of (intentional) misuse of p-values. The diagram displays science as objective, but there is also a great deal of subjectivity involved. $\endgroup$ Feb 17, 2023 at 13:53
  • 1
    $\begingroup$ @amr what is the mathematical method and what is pseudo-mathematics? $\endgroup$ Feb 17, 2023 at 13:54
  • 1
    $\begingroup$ Indeed, The Structure of Scientific Revolutions offers an alternative to Popper, but at a macrolevel of major theories, not the microlevel of looking at particular datasets, etc. $\endgroup$
    – Nick Cox
    Feb 17, 2023 at 14:00
  • 3
    $\begingroup$ I think we largely agree despite any indications otherwise. In my view there is scientific method which (e.g.) I try to apply but Donald J. Trump as far as I can tell does not. It implies e.g. respect for logic and evidence, without claiming that we (and certainly not I) have exclusive claim to any element. But there is limited value in trying to pin it down to a concise, precise, universal recipe, where limited does not mean zero. I dislike any emphasis on starting with observations. If a researcher starts anywhere it is with the state of knowledge in their field. $\endgroup$
    – Nick Cox
    Feb 17, 2023 at 14:12
  • 1
    $\begingroup$ @JaredSmith I am referring to said person's use of logic and evidence, which I consider a germane example. His political views are a different if related concern. $\endgroup$
    – Nick Cox
    Feb 17, 2023 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.