My question is rather semantic. When a method routinely produces high p-values it is called conservative. Would you call the opposite, i.e. a method with a high type-II-error rate liberal?
$\begingroup$
$\endgroup$
5
-
4$\begingroup$ Definitely not, if only because the term is so vague as to be meaningless. (I wouldn't use "conservative," either, without first explaining what I meant.) $\endgroup$– whuber ♦Commented Aug 6, 2018 at 12:51
-
2$\begingroup$ A method can be conservative without routinely producing high p-values. If the null is never true and you never take small samples the p-values may routinely be low but the method may still be conservative. Its opposite is sometimes called 'anticonservative' $\endgroup$– Glen_bCommented Aug 6, 2018 at 15:44
-
$\begingroup$ The cumulative probability of getting a p-value of p0 is, by definition, p0 (for the continuous case). Every method has the same chance of producing high p-values as every other method. $\endgroup$– AcccumulationCommented Aug 6, 2018 at 18:33
-
$\begingroup$ The term p-value is already doing enough damage when such concepts as 'conservative p-values' are around. P-values are used much too 'liberal' leading to all kinds of practices, conclusions and decisions which they should not. The p-value is a rule-of-thumb-measure of distance. When we wish to challenge a hypothesis then it is good to be 'conservative' (it makes us look further and better). I do not see the use of 'liberal' (too low) estimates of p-values unless we actually (a priori) believe it is good to reject the hypothesis and aim for power (which you dont necessarily get with liberal). $\endgroup$– Sextus EmpiricusCommented Aug 8, 2018 at 12:53
-
$\begingroup$ How about neurotic for a method that keeps producing high p-values $\endgroup$– AksakalCommented Aug 8, 2018 at 14:56
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
6
According to this homepage it is common to use this terminology.
Conservative in statistics has the same general meaning as in other areas: avoiding excess by erring on the side of caution. In statistics, “conservative” specifically refers to being cautious when it comes to hypothesis tests, test results, or confidence intervals. Reporting conservatively means that you’re less likely to be giving out the wrong information.
which can be specified in the following sense:
A conservative test always keep the probability of rejecting the null hypothesis well below the significance level. Let’s say you’re running a hypothesis test where you set the alpha level at 5%. That means that the test will (falsely) give you a significant result 1 out of 20 times. This is called the Type I error rate. A conservative test would always control the Type I error rate at a level much smaller than 5%, which means your chance of getting it wrong will be well below 5% (perhaps 2%).*
However I recommend you to use other terminologies, e.g. the definition of power. If a hypothesis test is "liberal" in your terminology it has more power. If a hypothesis test is "conservative" in your terminology it has less power. In my experience the term "a liberal hypothesis" is barely used in practice and might sound uncommon to your audience even if your audience consists of statisticians.
In the following paragraph I explain why "conservativ" and "liberal" are not always the exact difference in politics. Therefore I disrecommend using liberal as opposite of conservative in statistics. Feel free to ignore this part if it does not help you
Note that also in political science liberal is not necessarily the opposite of conservative. In the US left-wing politicians like Bernie Sanders are called liberals, but in many parts of Europe, e.g. Germany, the Netherlands and Denmark it is different. In German Politics Liberalism is mainly understand as the maximum of political freedom, especially in economics. The German Liberal Party (FDP) is in many issues rather right-wing than socialist even though they endorse issues like LGBT Rights and the Legalisation of Cannabis. Some Germans might think of what is called Libertarian in the US when you mention "liberal politics". In Denmark and the Netherlands it is even more complicated. You have two big parties which consider themself as liberal- In the Netherlands "VVD" and "D66"; In Denmark the "Vestre" and the "Radicale Vestre". While "VVD" and "Vestre" are rather "right-wing" the "D66" and the "Radicale Vestre" are rather left wing.
For this reason you should not use the terminology: "conservative statistical test" and "liberal statistical test" when speaking to a global, international audience.
PS: I hope I kept my political stance out of this topic and explained it neutrally.
-
2$\begingroup$ Semi-off-topic aside: "In the US left-wing politicians like Bernie Sanders are called liberals" – this is changing as well in the US. In at least certain wings, "establishment" Democrats are called "liberal" whereas more "radical"/"progressive" figures like Bernie, the Democratic Socialists of America, etc. are termed "left" but not "liberal." $\endgroup$– DanicaCommented Aug 6, 2018 at 13:46
-
2$\begingroup$ You could probably do without the section that describes the political usage of the words "conservative" and "liberal". It was somewhat distracting and not exactly on topic. $\endgroup$ Commented Aug 6, 2018 at 17:17
-
$\begingroup$ "A conservative test always keep the probability of rejecting the null hypothesis well below the significance level." That doesn't make sense. The probability of rejecting H0, given that it's true, is the significance level. $\endgroup$ Commented Aug 6, 2018 at 18:29
-
1$\begingroup$ @indigochild I separated my post in the "core answer" and an addendum. I hope this makes it more readible. $\endgroup$– FerdiCommented Aug 6, 2018 at 18:38
-
$\begingroup$ @Acccumulation. The goal is what you say, to design a test so the probability of rejecting H0, given that it's true, is the significance level. But this isn't always possible. A "conservative" test is designed so the probability of rejecting the null hypothesis is less than (or equal to) the significance level. $\endgroup$ Commented Aug 7, 2018 at 18:20
$\begingroup$
$\endgroup$
The question claims "when a method routinely produces high p-values it is called conservative." As pointed out by @Acccumulation in the comments, a p-value has a precise definition. One does not have more or less conservative p-values. In practice, sometimes one has to estimate a p-value (e.g. by using the bootstrap), and I suppose one could describe such an estimator as "conservative". But I haven't seen this in practice, and I don't think that's what the question is getting at.
Although I don't have a reference handy, it certainly seems natural to refer to one hypothesis test as being more conservative than another if it has a smaller type 1 error. Using liberal in the opposite sense seems possible, though I can't remembering seeing that anywhere.
The term "conservative" is often used for confidence intervals. A 95% confidence interval procedure will have different coverage probabilities depending on what the true value of the parameter is. For example, in Brown et al.'s Interval Estimation for a Binomial Proportion, speaking about two different confidence intervals for a Bernoulli probability p, they say "the coverage probability of the [Agresti–Coull] interval is quite conservative for p very close to 0 or 1. In comparison to the Wilson interval it is more conservative, especially for small n." Saying it's conservative for p very close to 0 or 1 means that for p close to 0 or 1, the probability of the interval containing the true value of p will be very high -- higher than the nominal coverage of the interval (say 95%).
