two sample analysis with outliers

Question

I've got some data from two groups which have different sample size. (univariate variable like 'price', and want to test whether they have significant difference )

The sample size of 'A' is 10,000 and sample size of 'B' is about 1,200.

I suspected outliers from data, so I used truncated data from each group for testing. (90 percent truncated value of group A : group_A_truncated0.9 = [percentile_A_5percent <= group_A_data percentile_A_95percent] )

But someone said that I have to get truncated value from using whole data (like truncated 90% value from [groupA + groupB]) for testing significance difference.

Am I wrong? plz help!

Answer 1

I wouldn't truncate at all, I would use a test other than the t-test. You could use a nonparametric test, or you could use a randomization test. You could also look at quantile quantile plots, parallel box plots and so on. You might even try quantile regression, with price as your DV and group as your IV.

You should remove outliers when they are errors - like a person who is 20 meters tall. When they are correctly entered, they are often interesting.

Answer 2

I fully agree with @PeterFlom's advice (+1) to be extremely cautious truncating outliers. Outliers should be ignored only if there is clear evidence that they are bogus--originating from equipment malfunction, data entry error, etc.

Even so, it is sometimes difficult to imagine that a sensible two-sample test of location is possible if one or both samples contain unusual values that seem to be 'outliers'. My purpose here is to explore alternatives to the two-sample t test.

Normal data. First, suppose we have 'uncontaminated' normal data. Sample 1 has $n_1 = 1000$ observations from $\mathsf{Norm}(\mu = 200, \sigma=25),$ rounded to three places. Sample 2 has $n_2 = 100$ observations from \mathsf{Norm}(\mu = 201, \sigma=25),$ similarly rounded. (Simulated sampling in R.)

 set.seed(2019)
 y1 = round(rnorm(1000, 200, 25), 3) 
 y2 = round(rnorm(100, 201, 25), 3)

Boxplots are shown below. The box for Group 2 is narrower because of the smaller sample size, 'notches' show nonparametric confidence intervals calibrated not to overlap if group populations are centered at the same value. As is typical of normal samples this large, there are some outliers. (Most, but not all, values in a normal sample tend to lie between $\mu \pm 2.75\sigma.)$

boxplot(y1, y2, col="skyblue2", pch=19, 
    varwidth=T, names=1:2, notch=T)
  abline(h=200, col="green2")

Both the Welch and the pooled two-sample t tests find that the group sample means 198.05 and 205.89 are significantly different at the 5% level:

t.test(y1, y2)

        Welch Two Sample t-test 

data:  y1 and y2
t = -3.2145, df = 123.52, p-value = 0.001668
 alternative hypothesis: true difference in means is not equal to 0
...
mean of x mean of y 
 198.0454  205.8917 

t.test(y1,y2,var.eq=T)$p.val   # pooled test
[1] 0.002701671

Also the nonparametric two-sample Wilcoxon (signed-rank) test finds that the two populations differ:

wilcox.test(y1, y2)$p.val
[1] 0.002317586

Data with many high 'outliers'. Now we simulate versions of the two samples that are contaminated with outlers:

set.seed(2019)
x1 = round(c(rnorm(1000, 200, 25), runif(20, 300, 500)),3)
x2 = round(c(rnorm(100, 201, 25), runif(2, 300, 500)),3)
boxplot(x1, x2, col="skyblue2", pch=19, varwidth=T, names=1:2)
  abline(h=200, col="green2")

The Welch two-sample t test (P-value 0.079) to longer finds a significant difference between the two samples (now with respective sample means 202.08 and 209.87) at the 5% level.

t.test(x1, x2)$p.val
[1] 0.07862595

However, the Wilcoxon test still finds a significant difference with P-value about 0.007.

wilcox.test(x1, x2)

       Wilcoxon rank sum test with continuity correction

data:  x1 and x2
W = 43624, p-value = 0.007129
alternative hypothesis: true location shift is not equal to 0

Additional possibilities for data with many ties. If your data are rounded to integers (or otherwise have many ties) and you do not trust your implementation of the Wilcoxon test to give valid a valid P-value, then you can do a welch t test on the ranks of the two samples combined. The ranks are not normal, but there can be no outliers among them. So when the sample sizes are as large as here, the t test on ranks seems within the robustness of a t test.

x = c(x1,x2);  g = c(rep(1,1020),rep(2,102))
t.test(x ~ g)$p.val
[1] 0.07862595     # same test and P-value as just above
t.test(rank(x) ~ g)$p.val
[1] 0.005733851    # significant P-value similar to Wilcoxon SR

A simulated permutation test is often another possibility when there are many outliers. But, depending on the 'metric' (measure of distrance) used, that kind of test often works best when sample sizes are roughly equal. (Otherwise, most data, and also most ties, get permuted to the larger sample--with subsequent lack of power.)

Example of a useful permutation test: However, if the metric to measure the distance between original and permuted samples is the difference in sample medians, then outliers play an appropriate role and we can make a productive simulation test. Here is R code for a permutation test using $m = 100,000$ random splits of all $n_1 + n_2 = 1122$ observations into a sample of size $1020$ and a sample of size $102$ and finding the difference in their medians.

There are over 28,000 distinct differences in medians. The simulated P-value of the permutation test is about $0.012 < 0.05,$ so the permutation test leads to rejection of the null hypothesis.

set.seed(624)
m = 10^5;  d.prm = numeric(m)
d.obs = median(x1) - median(x2);  d.obs
[1] -8.1185

for(i in 1:m) {
  x.p = sample(x)
  x.p1= x.p[1:1020]; x.p2 = x.p[1021:1122]
  d.prm[i] = median(x.p1)-median(x.p2) 
  }
mean(abs(d.prm) >= abs(d.obs))
[1] 0.01155               # P-value
length(unique(d.prm))
[1] 28459

The following histogram shows the simulated permutation distribution of the differences in medians. The probability outside the vertical lines at $\pm 8.1185$ is the P-value of the test.