1
$\begingroup$

I am interested in the holding cost distribution of a queuing system under different policies $\pi_i$ to see whether the theoretically optimal policy $\pi^*$ performs better in a statistically significant sense. As such, given the same simulator, I have sampled empirical distributions for two policies: $X_{\pi^*}$ and $X_{\pi_0}$. I consider the sample size $n=10000$ sufficiently large.

Due to the fact that holding costs are constrained to be positive, the distributions are skewed to the right. A Pearsons's and D'Agostino test has been applied to each distribution and has rejected the null hypothesis that they are normally distributed.

With each distribution stored as an array/vector, I have taken the element-wise difference between the two $\Delta X = X_{\pi^*} \ominus X_{\pi_0}$. Using the same pair of tests, we fail to reject that $\Delta X \sim \mathcal{N}(\mu,\sigma)$ where $\mu \leq 0$.

I would like to know if the mean performance of the optimal policy is statistically less than that of the other policy $\mu(X_{\pi^*}) < \mu(X_{\pi_0})$. The two non-normal distributions could be compared using a non-parametric test such as the Mann–Whitney U test. However, it would be nice to have an additional parametric test.

This leads us to the question: given that $\Delta X \sim \mathcal{N}(\mu,\sigma)$ *can we perform a student's t-test to test the null hypothesis that $H_0:\,\mu=0$ against the alternative $H_A:\,\mu<0$ *?

In other words, I am concerned that something is violated that I am missing. For example, is the element-wise subtraction valid or am I introducing some form of dependence?

With regards to other questions on Cross Validated, I consider this question to be the converse of this one. This one would suggest via the answer of @Glen_b that because the two distributions are skewed in the same direction a two-sample t-test would be biased and not robust such that I do not plan on taking this route.

Edit/Update:

I have added some histograms of the sampled distributions as well as their differences.

First, here are what the samples cost distributions of the two policies look like. Both failed the test for normality.

Cost distributions

The difference distribution $\Delta X $ follows below. I also present the result of my approach so far. Please do comment if any additional statistics are required.

differences distribution

Is distribution 1 better than distribution 2?
-------------------------------------------------
mean-difference:  -0.051223543253209096
sample-variance:  0.02349328805780243
theoretically better:  True
Normal distribution assumption: not rejected (p=0.0999)
Students t-test (alpha=0.05): 
>>> statistically significant: True (p=0.0)
Mann-Whitney U-test (alpha=0.05): 
>>> statistically significant: False (p=0.1394)

$\endgroup$
6
  • $\begingroup$ Is there some reason not to show a histogram and summary statistics for the differences? $\endgroup$
    – BruceET
    Commented Nov 18, 2021 at 18:07
  • $\begingroup$ @BruceET when the question was asked, the final simulations were being run. I based my question on preliminary small sample simulations with inadequate time-horizons. They are complete now and I can present an example. Thank you for this suggestion. $\endgroup$ Commented Nov 19, 2021 at 10:52
  • $\begingroup$ If you are doing a paired t test, that is precisely equivalent to a one-sample t test on the differences. So the paired test is valid if differences are normal. Just judging from your last histogram, the distribution seems to be close to normal with the alternative mean. You could make a normal QQ plot (normal probability plot) of the differences to see if that plot is essentially linear (perhaps except for a few wobbles in the extreme tails). You could also superimpose a normal dist'n (with mean and SD matching the sample $\bar X$ and $S)$ to see how close it comes to the histogram. $\endgroup$
    – BruceET
    Commented Nov 19, 2021 at 16:15
  • $\begingroup$ @BruceET Thank you very much for your reply. I am doing a two-sample t-test as the data is not paired. The difference distribution resulted from element-wise subtraction between the two arrays of samples. Is this perhaps an incorrect thing to do if one is not doing a paired-test? After all, the difference between two random variables is done through convolution. Perhaps the difference distribution should be generated through convolution of the two arrays instead. $\endgroup$ Commented Nov 19, 2021 at 16:27
  • 1
    $\begingroup$ Those histograms look pretty normal to me. Keep in mind that normality testing is less helpful than one might hope. $\endgroup$
    – Dave
    Commented Nov 19, 2021 at 17:32

1 Answer 1

1
$\begingroup$

I am not sure how you are generating your two original skewed samples, or your purposes in looking at the differences in means. So what follows is just an illustration of my Comment--modified in view of your replies to my comment.

Because I do not understand your exact objectives, working with simulated data in this way, I am suggesting possible approaches, not recommending an exact course of action.

Numerical and graphical descriptions of data. My x1 and x2 are not correlated. So, as you say in your reply, you should use a 2-sample test. In that case, I do not see the point in looking at a plot of the differences; for example, the 2-sample t test looks at a difference in the two sample means together with a combined variance estimate.

set.seed(1119)
x1 = rgamma(1000, 5, .1)
x2 = rgamma(1000, 5, .09)

As appropriate to these gamma populations, sample mean are about $\mu_1 = 50$ and $\mu_2 = 55.5.$ respectively. The independently sampled x1 and x2 are uncorrelated with a sample correlation near $0.$

summary(x1); sd(x1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   6.26   34.60   45.68   49.22   61.19  173.13 
[1] 22.17256  # SD first sample

summary(x2); sd(x2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   8.41   36.88   50.89   55.24   68.62  167.12 
[1] 25.2172

cor(x1, x2)
[1] 0.01881356

These samples from gamma distributions have similar shapes (moderately right skewed), as shown in the boxplots. The scatterplot shows no association.

Histograms of the individual samples also show this skewness. The density curves (blue) are for $\mathsf{Gamma}(5,.1)$ and $\mathsf{Gamma}(5,.09),$ respectively. The red density curves are for normal distribution matching the means and SDs of the two samples.) It is clear that the samples are mildly skewed.

enter image description here

R code for figure:

par(mfrow=c(2,2))
boxplot(x1, x2, col="skyblue2", pch=".")

plot(x1, x2, pch=".")
hist(x1, prob=T, ylim=c(0,.02), col="skyblue2")
 curve(dgamma(x,5,.1), add=T, col="blue")
 curve(dnorm(x,mean(x1),sd(x1)), add=T, col="red")
hist(x2, prob=T, ylim=c(0,.02), col="skyblue2")
 curve(dgamma(x,5,.09), add=T, col="blue")
 curve(dnorm(x,mean(x2),sd(x2)), add=T, col="red")
par(mfrow=c(1,1))

Here is a normal probability plot of the first sample. It is clearly not linear.

qqnorm(x1)
 qqline(x1, col="red")

enter image description here

Two-sample tests. Some statisticians would argue that a two-sample t test should be sufficiently robust against mild skewness for samples as large as 1000. For data similar to those shown here, I would prefer using a two-sample Wilcoxon rank sum test for difference in location. As shown below, for my fictitious data, both tests show highly significant results.

t.test(x1,x2)

       Welch Two Sample t-test

data:  x1 and x2
t = -5.6702, df = 1965.8, p-value = 1.637e-08
alternative hypothesis: 
 true difference in means is not equal to 0
95 percent confidence interval:
 -8.103426 -3.938478
sample estimates:
mean of x mean of y 
 49.22117  55.24212 

wilcox.test(x1, x2)

        Wilcoxon rank sum test 
        with continuity correction

data:  x1 and x2
W = 432070, p-value = 1.439e-07
alternative hypothesis: 
 true location shift is not equal to 0
$\endgroup$
1
  • 1
    $\begingroup$ This has been a enjoyable read. Thank you for the time taken to add intuitive understanding to your answer. I also appreciate the additional Wilcoxon rank sum test as well as testing the samples for correlation (a step that I had originally omitted and could have lead erroneous reporting). $\endgroup$ Commented Nov 20, 2021 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.