Applying a non-normality and non-homoscedasticity penalty to the p-value of a t-test

Question

【My motivations】:(Add on 05/10/2019 05:23 (JST).)

In the traditional textbook, method like following codes are written. (Since our purpose is not to explain what is written in the textbook, the one corresponding to Code 01 below is omitted.)

Code 02':

Group_A = c(40.2, 40.4, 40.6, 40.8, 41.0, 41.2, 41.4, 41.6, 41.8)
Group_B = c(30.1, 30.3, 30.5, 30.7, 30.9, 31.1, 31.3, 31.5, 31.7, 31.9, 32.1)

p1A=shapiro.test(Group_A)
p1B=shapiro.test(Group_B)
p1A
p1B

p2=var.test(Group_A,Group_B)
p2

teqv=t.test(Group_A,Group_B, var.equal = T)
teqv

The comments like followings are written on the books describing the method like code 02': If we can deny non-normality and we can deny that the 'variances are not equal', perform t-test But isn't this a multiple test? If so, isn't it necessary to correct the P value?

On the other hands, some books describe a method like code 02'' below.

Code 02'':

Group_A = c(40.2, 40.4, 40.6, 40.8, 41.0, 41.2, 41.4, 41.6, 41.8)
Group_B = c(30.1, 30.3, 30.5, 30.7, 30.9, 31.1, 31.3, 31.5, 31.7, 31.9, 32.1)


twel=t.test(Group_A,Group_B, var.equal = T)
twel

But, how can we justify the normality and equal variance of the population? (Did they escape from the evaluation that should be done to avoid the problem of multiple testing?)

From the aspect of p-value definition:
There seem to have been various opinions about the definition of p-value. Recently, many editorials criticizing traditional statistics has been "mass-produced" in leading journals including Nature. The definition of p-value is ambiguous even though so many editorials have been forged. The following is the full text from the ASA statement in the chapter named “What is a p-Value?”. It's hard to understand that a fine organization like ASA can only write “informal” definitions to criticize the misuse of p-values. But unfortunately, this was the clearest as far as I know.

What is a p-Value?
Informally, a p-value is the probability under a specified statistical model that a statistical summary of the data (e.g., the sample mean difference between two compared groups) would be equal to or more extreme than its observed value.

The ASA statement also has the following description in the other section.

P-values can indicate how incompatible the data are with a specified statistical model.

A p-value provides one approach to summarizing the incompatibility between a particular set of data and a proposed model for the data. The most common context is a model, constructed under a set of assumptions, together with a so-called “null hypothesis.” Often the null hypothesis postulates the absence of an effect, such as no difference between two groups, or the absence of a relationship between a factor and an outcome. The smaller the p-value, the greater the statistical incompatibility of the data with the null hypothesis, if the underlying assumptions used to calculate the p-value hold. This incompatibility can be interpreted as casting doubt on or providing evidence against the null hypothesis or the underlying assumptions.

They don't say that the t-test itself is/is not right way, but according to the P-values on above-mentioned sense, the P-value shall be an indicator of "how unusual is the particular set of data observed under the assumption is correct". As the incompatibility between a particular set of data and a proposed model for the data, is it right way to ignore the effects of normality and equal variances? In characterizing the proposed model (model, constructed under a set of assumptions, together with a so-called “null hypothesis.”), normality and equality of variance should be as important as the null hypothesis. In this sense, I feel that the t-test should also be renewed.

I think that the correct t-test in the following Experiment should be calculated as shown in Code 01 and 02 below.

However, I have never seen any document having such description.

【My Question】

• Is my way of thinking (the method of the following code01 and code02) correct?
• If my way of thinking is incorrect, why isn't there a way to reflect the factors that affect truth in the p-value?

If the normality is high, even if the “p-value in the original sense” is high, the “p-value I say” goes down. However, I am asking questions after understanding that it is strange. I would like you to point out where my logic is broken.

Experiment:
We randomly divide 20 animals into two groups, Group A and Group B. After that, for Group A, Foods A are fed, and for Group B, Foods B are fed. After a certain period, bodyweight was measured, and the data were as follows.

Group_A :40.2, 40.4, 40.6, 40.8, 41.0, 41.2, 41.4, 41.6, 41.8
Group_B :30.1, 30.3, 30.5, 30.7, 30.9, 31.1, 31.3, 31.5, 31.7, 31.9, 32.1

I would like to conduct a two-sided test with a significance level of 0.05 to see if there is a significant difference between the two groups.

You can download Following Codes from this site.

Code01

Group_A = c(40.2, 40.4, 40.6, 40.8, 41.0, 41.2, 41.4, 41.6, 41.8)
Group_B = c(30.1, 30.3, 30.5, 30.7, 30.9, 31.1, 31.3, 31.5, 31.7, 31.9, 32.1)

p1A=shapiro.test(Group_A)
p1B=shapiro.test(Group_B)

twel=t.test(Group_A,Group_B, var.equal = F)
twel

ans = p1A$p.value * p1B$p.value * twel$p.value
ans

Code 02

Group_A = c(40.2, 40.4, 40.6, 40.8, 41.0, 41.2, 41.4, 41.6, 41.8)
Group_B = c(30.1, 30.3, 30.5, 30.7, 30.9, 31.1, 31.3, 31.5, 31.7, 31.9, 32.1)

p1A=shapiro.test(Group_A)
p1B=shapiro.test(Group_B)

p2=var.test(Group_A,Group_B)

teqv=t.test(Group_A,Group_B, var.equal = T)
teqv

ans = p1A$p.value * p1B$p.value * teqv$p.value * p2$p.value
ans

Why I think so.
The reason why I think so is that it is considered that all assumptions of the t-test should be reflected in the p-value.

Actually, the probability that three independent events, event1, event2 and event3 are correct at the same time ($P(E1\cap E2\cap E3)$) is represented by following equation. Here, P(Ei) represents the probability that event 1 is correct

$$P(E1\cap E2\cap E3) = P(E1)P(E2)P(E3)$$

“Factors that affect truth in the p-value” shall be “normality of both groups” in the case of Welch's test (see code01). If we assume that the variances of both groups are equal, then “equal dispersion” is added to this(see code 02). The probability that these are true seems to be independent.

Ref:
https://en.wikipedia.org/wiki/Shapiro%E2%80%93Wilk_test

Answer 1

Generally speaking, I believe that best practice is to do a Welch 2-sample t test without using the test of equal variances to 'branch' to a pooled test or a Welch test. Using the variance test to decide between the two 2-sample t tests has been deprecated; see Comment by @whuber about difficulties interpreting multiple P-values.

Also, if boxplots show no far outliers or extreme skewness, any departure from normality for samples of this size should be accommodated by the 'robustness' of the t test.

For your data, there seems little doubt that the two groups differ. All Group A observations exceed any Group B observation.

Group_A = c(40.2, 40.4, 40.6, 40.8, 41.0, 41.2, 41.4, 41.6, 41.8)
Group_B = c(30.1, 30.3, 30.5, 30.7, 30.9, 31.1, 31.3, 31.5, 31.7, 31.9, 32.1)
t.test(Group_A,Group_B)

        Welch Two Sample t-test

data:  Group_A and Group_B
t = 36.558, df = 17.993, p-value < 2.2e-16
alternative hypothesis: 
  true difference in means is not equal to 0
95 percent confidence interval:
  9.331051 10.468949
sample estimates:
mean of x mean of y 
     41.0      31.1 

boxplot(Group_A, Group_B, notch=T, col="skyblue2")

The boxplots below show the complete separation of A and B values. The 'notches' in the sides of the boxes are nonparametric confidence intervals for population medians, calibrated for comparing two groups, so that non-overlapping confidence intervals suggests a significant difference in medians.

Furthermore, there are ${20 \choose 9} = 167\,960$ ways in which to apportion 20 scores into two groups of 9 and 11; only $2$ of them with complete separation of scores. So a 2-sided permutation test has P-value 1.19076e-05.

choose(20, 9)
[1] 167960
2/choose(20, 9)
[1] 1.19076e-05