2
$\begingroup$

I'm seeking a non-technical explanation of how to interpret the Hill estimate of the tail index for fat-tailed data, and, if possible, some explanation of seemingly contradictory results that different R functions produce.

I have watched some tutorials of tail index estimation and extreme value theory (e.g., https://www.youtube.com/watch?v=GZPqQPQQZAk&t=18s, https://www.youtube.com/watch?v=o-cpu1IH3tM) and have been looking through the documentation of R functions for calculating the Hill estimate of the tail index (namely, evir::hill and ReIns::Hill.kopt), but am having trouble making sense of results I get.

Here's an example where I pose more concrete questions:

Step 1

First, I load the relevant packages and create three log normal distributions with varying tail lengths, dat_long, dat_longer, and dat_longest.

# packages
library(ReIns)
library(evir)
library(tidyverse)
library(cowplot)

set.seed(42)

# create three distributions with long, longer, and longest tails
dat_long <- rlnorm(n=500, meanlog = 0, sdlog = 1)
dat_longer <- rlnorm(n=500, meanlog = 0, sdlog = 10)
dat_longest <- rlnorm(n=500, meanlog = 0, sdlog = 20)

# view distributions
a <- ggplot()+geom_histogram(aes(x=dat_long))
b <- ggplot()+geom_histogram(aes(x=dat_longer))
c <- ggplot()+geom_histogram(aes(x=dat_longest))
cowplot::plot_grid(a, b, c, ncol = 1)

enter image description here

Step 2

Then I use evir::hill to generate Hill plots for each distribution.

par(mfrow=c(3,1))
hill(dat_long)
hill(dat_longer)
hill(dat_longest)

enter image description here

Step 3

Now I need to select the optimal threshold. In this tutorial it's explained that you should look for where the plot stabilises. So eyeballing the above plots, I'd conclude that the optimal order statistic (or k) for dat_long would be around 145 and the alpha would be about 1.5. For dat_longer the optimal order statistic would be around 189 and alpha would be about 0.15. For dat_longest the optimal order statistic would be around 93 and alpha would be about 0.10.

From this I would conclude that, as a general rule of thumb, the Hill estimate decreases as tail length increases. Is this a valid conclusion?

Step 4

Since I'm not really confident with the "eyeballing" method, I turn to ReIns::Hill.kopt, which as the documentation states selects the optimal threshold for the Hill estimator by minimising the Asymptotic Mean Squared Error (AMSE). This function outputs the optimal order statistic as kopt and the optimal Hill estimate as gammaopt.

So I apply it each distribution and get the following:

> Hill.kopt(dat_long)$kopt
[1] 60
> Hill.kopt(dat_long)$gammaopt
[1] 0.4077617
> 
> Hill.kopt(dat_longer)$kopt
[1] 92
> Hill.kopt(dat_longer)$gammaopt
[1] 5.167227
> 
> Hill.kopt(dat_longest)$kopt
[1] 1
> Hill.kopt(dat_longest)$gammaopt
[1] 0.03570751

Why do these results have no resemblance to the results I arrived at after eyeballing the Hill plots with evir::hill?

Step 5

Given the weird results in Step 4, I checked if there was any relationship between the optimal Hill estimate returned by Hill.kopt and tail length (in this case, sdlog). As the plot below shows, it seems like the answer is no.

# calculate optimal Hill estimate for distributions 
# with sdlog = {1,2,3,...20}. i.e., varying tail length
hills <- NULL
for(i in 1:20){
  dat <- rlnorm(n=500, meanlog = 0, sdlog = 1)
  h <- Hill.kopt(dat)$gammaopt
  hills <- rbind(hills, c(i, h)) %>% as.data.frame()
}

# plot optimal Hill estimate (gamma) over tail length (sdlog)
names(hills) <- c("sdlog", "gamma")
ggplot(hills, aes(x=sdlog, y=gamma))+
  geom_point(size = 3, alpha = 0.7)+
  geom_smooth(method = "lm")+
  theme_minimal()+
  labs(y = "Hill Estimate (gamma)",
       x = "Tail Length (sdlog)")

enter image description here

How should I interpret/reconcile these results?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.