Should the Wilcoxon Rank sum test be used for testing the mean difference significance?

Question

I was looking replicate the results of the paper DOI:10.3905/jpm.2014.40.3.087 (Exploring Macroeconomic Sensitivities: How Investments Respond to Different Economic Environments, Ilmanen Maloney Ross 2014 Journal of Portfolio Management).

For example, I was looking to investigate if the monthly total return of US Treasury bond index (LUATTRUU Index - Bloomberg Barclays US treasury total return index) depends on the magnitude of the US inflation readings (CPI YOY Index - US CPI Urban Consumers YoY NSA »). So i downloaded the US CPI YOY monthly data for the last 50 years and the monthly total returns of US Treasury index for the same period. Then, I've partitioned the sample in "UP" if the monthly US inflation readings is above its last 12 monthly average, and "DOWN" if the opposite happened. Then, i computed the average monthly total return of US Treasury bond index in each subsample. I got the following results.

1. US CPI "UP" -> monthly avarage US Treasury bond index total return 0.40%
2. US CPI "DOWN" -> monthly avarage US Treasury bond index total return 0.65%

So, the data seems to confirm that when inflation is high, bond return are low compared to the opposite situation, as the economic theory would suggest.

I would like to test statistically if the difference in the two return is statistically significant. However, since i'm using financial time series, I'm not sure which statistical test should I use.

Here my R code, that takes as input the time series of US CPI YOY and US TREASURY TOTAL RETURN INDEX:

CPI_YOY_roll<- rollmean(CPI_YOY,k=12, fill=NA, align="right")
a<-ifelse(CPI_YOY[13:nrow(ds)]>=CPI_YOY_roll[12(nrow(ds)-1)],"up","down")
mean(na.omit(ifelse(a=="up",ds$treasury_return,NA)))*100
mean(na.omit(ifelse(a=="down",ds$treasury_return,NA)))*100

For example, here Ilmanen Maloney Ross (2014, ExploringMacroeconomic Sensitivities: How Investments Respond to Different Economic Environments, Journal of Portfolio Management) use my same procedure and get the following results (using average Sharpe ratio instead of average return, but the concept is the same):

So i want to test if these differences are statistically significant or not!

Answer 1

Consider using a regression model with a test of slope parameter instead

While a T-test should serve adequately for testing for a mean difference between two groups, I would counsel against the method you are using here. In particular, there is a substantial loss of information caused by converting the inflation rate into a simple up/down binary. A secondary problem with this method is that your up/down conversion is taken relative to the previous 12-month average, which then induces a dependency in this binary variable on the previous inflation rates. The resulting comparison does not appear to me to capture your initial research hypothesis of interest.

If you want to know how the monthly total return of US Treasury bond index depends on the magnitude of the US inflation readings, I recommend modelling these variables directly against each other without categorising them. This analysis can be done with a regression model and estimation of the relationship can be done using a standard test on the slope of the regression line.

Your question does not specify which particular measures you have downloaded as your data, so I will show an example of the analysis using some data I have chosen instead. In the example below I have measured CPI with the CPALTT01USM661S (Consumer Price Index: All Items: Total for United States, Index 2015=100, Monthly, Seasonally Adjusted) and I have measured treasury yield with DGS10 (Market Yield on U.S. Treasury Securities at 10-Year Constant Maturity, Quoted on an Investment Basis, Percent, Monthly, Not Seasonally Adjusted), with both variables available on a monthly basis from the start of 1962 up to late 2023 (741 months available). I have converted the measurements to "force of interest" using logarithmic changes and used a simple linear regression model to relate these variables.

#Load the data
DATA <- read.csv('Economic Data.csv')

#Create variable for annualised inflation rate (percentage)
n <- nrow(DATA)
DATA$BondRate <- log(1 + DATA$BondYield/100)
DATA$Inflation <- 0
DATA$Inflation[2:n] <- 12*diff(log(DATA$CPI))
DATA <- DATA[-1, ]
n <- nrow(DATA)

#Model the data
MODEL <- lm(BondRate ~ Inflation, data = DATA)

#Plot the data
plot(100*DATA$Inflation, 100*DATA$BondRate, 
     xlab = 'Inflation Rate (%)',
     ylab = '3-Month T-Bill Rate (%)')
abline(a = 100*MODEL$coefficients[1], b = MODEL$coefficients[2], lty = 3)

We can test whether there is a relationship between the variables by looking at the estimated slope coefficient in the model. In the summary below we see that there is strong evidence of a non-zero slope coefficient, which means that there is strong evidence of a relationship between the variables.

#Show model summary
summary(MODEL)

Call:
lm(formula = BondRate ~ Inflation, data = DATA)

Residuals:
      Min        1Q    Median        3Q       Max 
-0.062104 -0.016747 -0.000591  0.013270  0.086906 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 0.045700   0.001336   34.21   <2e-16 ***
Inflation   0.292355   0.024964   11.71   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.02577 on 738 degrees of freedom
Multiple R-squared:  0.1567,    Adjusted R-squared:  0.1556 
F-statistic: 137.2 on 1 and 738 DF,  p-value: < 2.2e-16

Answer 2

There are a variety of complicated methods you could use to test this, but I think one that balances simplicity, with accounting for the autocorrelation in financial time series, would be to use generalized least squares. Here you would have a single explanatory variable which is 0 or 1 based on if you are in the US CPI "UP" or US CPI "DOWN" category, and your response variable would be your returns (although you may want to use log returns). You would fit your GLS model and then see if the $\beta$ associated with your binary variable is statistically significant.

More information about this method can be found in Introduction to Time Series Forecasting by Brockwell and Davis page 212, as well as in Appendix B.1.2. Also see the section Feasible Generalized Least Squares in this Wikipedia article. I would suggest modelling your residuals as a AR(1) or AR(2) model.

I will also note that the problem with that Wilcoxon rank-sum test for your problem is you need to assume the samples are independent, but because of the autocorrelation in your time series this assumption is violated.