I'm seeking a non-technical explanation of how to interpret the Hill estimate of the tail index for fat-tailed data, and, if possible, some explanation of seemingly contradictory results that different R functions produce.
I have watched some tutorials of tail index estimation and extreme value theory (e.g., https://www.youtube.com/watch?v=GZPqQPQQZAk&t=18s, https://www.youtube.com/watch?v=o-cpu1IH3tM) and have been looking through the documentation of R functions for calculating the Hill estimate of the tail index (namely, evir::hill
and ReIns::Hill.kopt
), but am having trouble making sense of results I get.
Here's an example where I pose more concrete questions:
Step 1
First, I load the relevant packages and create three log normal distributions with varying tail lengths, dat_long
, dat_longer
, and dat_longest
.
# packages
library(ReIns)
library(evir)
library(tidyverse)
library(cowplot)
set.seed(42)
# create three distributions with long, longer, and longest tails
dat_long <- rlnorm(n=500, meanlog = 0, sdlog = 1)
dat_longer <- rlnorm(n=500, meanlog = 0, sdlog = 10)
dat_longest <- rlnorm(n=500, meanlog = 0, sdlog = 20)
# view distributions
a <- ggplot()+geom_histogram(aes(x=dat_long))
b <- ggplot()+geom_histogram(aes(x=dat_longer))
c <- ggplot()+geom_histogram(aes(x=dat_longest))
cowplot::plot_grid(a, b, c, ncol = 1)
Step 2
Then I use evir::hill
to generate Hill plots for each distribution.
par(mfrow=c(3,1))
hill(dat_long)
hill(dat_longer)
hill(dat_longest)
Step 3
Now I need to select the optimal threshold. In this tutorial it's explained that you should look for where the plot stabilises. So eyeballing the above plots, I'd conclude that the optimal order statistic (or k) for dat_long
would be around 145 and the alpha would be about 1.5. For dat_longer
the optimal order statistic would be around 189 and alpha would be about 0.15. For dat_longest
the optimal order statistic would be around 93 and alpha would be about 0.10.
From this I would conclude that, as a general rule of thumb, the Hill estimate decreases as tail length increases. Is this a valid conclusion?
Step 4
Since I'm not really confident with the "eyeballing" method, I turn to ReIns::Hill.kopt
, which as the documentation states selects the optimal threshold for the Hill estimator by minimising the Asymptotic Mean Squared Error (AMSE). This function outputs the optimal order statistic as kopt
and the optimal Hill estimate as gammaopt
.
So I apply it each distribution and get the following:
> Hill.kopt(dat_long)$kopt
[1] 60
> Hill.kopt(dat_long)$gammaopt
[1] 0.4077617
>
> Hill.kopt(dat_longer)$kopt
[1] 92
> Hill.kopt(dat_longer)$gammaopt
[1] 5.167227
>
> Hill.kopt(dat_longest)$kopt
[1] 1
> Hill.kopt(dat_longest)$gammaopt
[1] 0.03570751
Why do these results have no resemblance to the results I arrived at after eyeballing the Hill plots with evir::hill
?
Step 5
Given the weird results in Step 4, I checked if there was any relationship between the optimal Hill estimate returned by Hill.kopt
and tail length (in this case, sdlog
). As the plot below shows, it seems like the answer is no.
# calculate optimal Hill estimate for distributions
# with sdlog = {1,2,3,...20}. i.e., varying tail length
hills <- NULL
for(i in 1:20){
dat <- rlnorm(n=500, meanlog = 0, sdlog = 1)
h <- Hill.kopt(dat)$gammaopt
hills <- rbind(hills, c(i, h)) %>% as.data.frame()
}
# plot optimal Hill estimate (gamma) over tail length (sdlog)
names(hills) <- c("sdlog", "gamma")
ggplot(hills, aes(x=sdlog, y=gamma))+
geom_point(size = 3, alpha = 0.7)+
geom_smooth(method = "lm")+
theme_minimal()+
labs(y = "Hill Estimate (gamma)",
x = "Tail Length (sdlog)")
How should I interpret/reconcile these results?