There are some copula families that are able to deal with upper tail dependence (Gumbel and Joe copula) or lower tail dependence (Clayton). We can also rotate these copulae to have another copula. For example, if we rotated Gumble copula by 90 degrees then we will have a new copula that is able to describe negative tails (at the corner [0,1]). I found that there is no tail dependence for this copula. From VineCopual package I found the tail dependence of this copula is zero. However, I can see from the plot there is a negative upper tail dependence. My question is, how can I describe this type of dependence?
What you seem to have done is
- generate data $(U_i,V_i)$ from a Gumbel copula.
- rotate your copula by considering $(U_i^*,V_i)=(-U_i,V_i)$
Now, computing the (upper) tail-dependence is about restricting ourselves to the top-right corner, i.e. looking at both upper-tails. Hence it is not surprising to obtain a null upper-tail dependence.
What you are interested about, I believe, happens in the top-left corner: you are looking at the lower-tail of $U^*$ jointly with the upper-tail of $V$, which is indeed equivalent to looking at the upper-tail of $(U,V)$.
There might be something available (in R) that specifically compute the tail-dependence in that corner (top-left), but I do not see why you could not simply compute the regular upper-tail dependence $(-U^*,V)$ directly (which amounts to $(U,V)$ in the example).