question about p values and significance comparison [duplicate]

Question

I am under the impression that you have to set your alpha value before the experiment, and once the experiment is done, your p value, based on the alpha value, is either significant or not. It's dichotomous. There is no "almost significant" or anything like that.

So if I set my alpha at 0.04, but my p value is 0.05 then my results are insignificant.

I do not understand why. Imagine a beginner researcher who really wants to get significant result, setting alpha at 0.2 and gets 0.19 p so its significant. Or imagine a researcher so sure that a treatment works that sets alpha at 0.001 but his p value is 0.002 so its insignificant. Why has the significance of the result have anything to do with the researcher's guess about alpha levels?

Why can't we treat p value as a continuous variable, instead of randomly picking some alpha significance level before we start?

Answer 1

The purpose of setting a threshold is to remove post hoc fudging and provide clarity in the decision making process. This is not to say this is how it is actually used, but it is the ideal behind it.

The thinking behind the process is that you decide how much risk you are willing to take on missing a real effect of a size that matters to you (powering the study on the assumption of an expected effect size) versus the risk of believing data that would not be distinguishable from no effect if the experiment were repeated a vat number of times.

This is meant to force you to design the whole experiment, including analysis and decision making up front. If you can't do this, then you are probably in exploratory analysis phase and relying on crude rules of thumb, in which case any outcomes that appear interesting should be validated in independent experiments.

Why has the significance of the result have anything to do with the researcher's guess about alpha levels?

The alpha shouldn't be a wild guess, rather an educated one. It is about managing risk and should be matched with appropriate powering procedures during design to be most useful.

Why can't we treat p value as a continuous variable, instead of randomly picking some alpha significance level before we start?

This is reasonable, as long as there are clear rules about how it is handled. For example exploratory analysis may rank p values and filter the top x percent for further testing. I've used this approach to filter principal components for discriminant analysis. Another alternative is to have the post hoc decision tied to the value of p, so one example could be if the results are extremely strong a full R&D budget could be rubber stamped to continue development but is tapered off over a range of p values. In a major multinational R&D intensive organisation this would help balance risk across a research portfolio

The example you give is inappropriate as you want to fudge interpretation once you know your data. This is one aspect of investigator bias.

Clear logical rules need decided before you see the data so as not to bias interpretation. The rules don't have to be the standard rules of thumb or threshold, but they need clearly and unambiguously explained and be logically consistent.

Answer 2

This issue is very controversial. I imagine that this answer will have many negative votes. It has implications for the daily lives and work of many. But you have to do what you have to do.

That's more or less how I understand the story:

1. 100 years ago Fisher developed a method for data testing.
2. Neyman and Pearson took what Fisher developed and developed it further.
3. The two proposals are similar but contradictory on some points.
4. Since the proposals were published, there have been publications that questioned them.
5. Towards the 1940s or 1950s (a task for historians), the two theories were almost anonymously combined into a single theory now called NHST (null hypothesis statistical tests).
6. No one is the author of the NHST. It's one more rule of thumb. So you can't discuss this theory with any author.
7. Since their anonymity, NHSTs have been criticized. There are hundreds (or thousands) of articles indicating that they are not the most appropriate method of doing scientific research.
8. Last year the ASA in its conference "a world beyond p<0.05) seems to have started the abandonment of NHSTs.

NHST problems:

1. Logical: based on failed logical tests.
2. Real: It is based on ideal distributions, which are rarely found in reality.
3. They are of anonymous creation: With which author can you argue?.
4. They give a false sense of security. The problem of inference is hundreds or thousands of years old. Philosophers still can't figure it out. We do?

Researcher positions:

1. Abandon NHSTs and use Bayesian statistics. Problem: It is not a total solution because it would have the same problem as the NHST, in some cases reasoning in the same way.
2. Use NHSTs well. Problem: 5 books will explain NHSTs in 5 different ways, because there is no established method for making NHSTs. NHSTs were invented by us, there's no evidence on how to evaluate the evidence!
3. Create new methods for data testing. That's where some of the authors are.

Answer 3

Interesting question philosophically. I will give and answer, but I may be corrected mathematically.

The idea is that you go into the experiment with a null hypothesis, or as my statistics professor used to say, "the boring and uninteresting result." SO, if you are trying to prove that smoking leads to earlier death, the null hypothesis is "smoking does not lead to earlier death", which means that the average age of a smoker is the same as a non-smoker if you draw them randomly from the real world.

The statistics is about proving that the "boring" result does not appear to hold, because the odds of that happening given the data you looked at makes it highly unlikely it was just due to randomness that those averages are not close enough. Think of it this way: if you had exactly one smoker and non-smoker, you really don't have enough data to make such a determination (especially if the non-smoker was killed by a bus). But, if you have 10 million of each, you can have a lot of accuracy (or confidence), statistically speaking, if you don't get averages that are close to each other.

I believe the alpha logic comes from coming into it. The question is, a priori, what level of confidence do you expect (or would be acceptable) to make you believe the null hypothesis was wrong? For one thing, that keeps you from designing the experiment in such a way that what you truly believe in your heart (and is consistent with the experiment you set up) is that "95% would really be convincing" doesn't lead to your analyzing the data, getting 92%, and then saying "90% makes it true".

The level of significance changes by field. In some areas of particle physics, for example, where they record trillions of 'events' looking for something elusive, the standard is 5- or 6-sigma. Otherwise, you will find false positives all the time given the number of events. That is an example of how the experiment plays into it.

On the other hand, when I did business consulting, we often presented regressions with 75% p-values. The logic there is that companies must make a lot of decisions, and a lot of decisions that have a p-value of 75% that get done quickly (and can be reversed if they prove wrong) is a lot more useful than proving a small number of things at 99% confidence.

I'm not sure this is entirely helping you or answering what you are asking, but in practice (especially in a field like economics) people sort of do treat it as continuous. Any variables that come out of regression with a p-value of 95% feel really strong; any above 90% seem quite strong; but you wouldn't ignore a variable at 89%. In economics, one issue is that you don't really design experiments, and data can be a bit 'dirty' itself.

Maybe another way to see this is lime this: suppose you are testing a diagnostic screening against a disease that is surely fatal, but the treatmen itself kills 10% of the people who get it whether or not they have the disease, but saves 100% of the people who have the disease (imagine Ebola, or AIDS in the early years). But suppose only 0.1% (one in 1,000) of the population is infected. If your test gives 5% false positives, and perfect actual positives, that means out of every 1000 people tested 1 will have the disease but 51 will test positive (the one real and 50 false). That means the drug will kill, out of every 1000 people tested, 5.1 people - only one of whom had the disease. Therefore, it would be better not to use you test because without it there will only be one death. Clearly, in that case, you would come in needing a much 'tighter' alpha a priori to justify proving that using the test is a good idea.

That's all a bit sloppy technically, but I think it gets at what you are asking.