- I have been able to derive a matrixsquare matrix to analyse 1° of overlap (ie. #items overlapping per pair of categories) following the advise in Category Overlap Analysis question .
- unpivoting the matrix to derive an outputoutput that can bewas used to analyse overlaps per pair of categories and identify duplication that exists for 1° overlaps. Through this analysis I am able to identify 300 duplicated 1° categorical overlaps which can be resolved to 118 unique categories.
I have also attempted to use visualisations techniques to achieve the outcome, but to no avail:
- Chord Diagram: there is 1200+ categories, the limit to chord diagram is 400+
- UpSet with this json: but can't to configure it to derive the insights needed.
Question:
Duplicated category will be biased towards category ID with smaller #. (ie. Where same categories are exactly the same, the category with higher ID will be marked as a duplicate of the category with the smallest ID. This example, G0419 and G0846 are both marked as duplicates of G0080)
Modified overlap function to identify category that is fully represented by its subsumed categories and remove subsumed relationship relating to identical categories. new plot
overlaps = function(ta,row=T,sub=F){ ## Takes a dichotomous table and computes the overlap
## init variables ----
if(row==F){ta=t(ta)}
ncas = dim(ta)[1]
outmat = matrix(0,nrow=ncas,ncol=ncas)
ilist = list()
slist = list()
## loop through rmat to establish equality & subsumation ----
for(n1 in 1:ncas){View
ilist[[n1]]=c(NA)
slist[[n1]]=c(NA)
for(n2 in 1:ncas){
if(n1!=n2){
d = ta[n1,]-ta[n2,]
if(min(d)==max(d)){
outmat[n1,n2]=1 ## Equality
t_list<- c(ilist[[n1]],n2,n1)
ilist[[n1]]= t_list[which.min(t_list)]
}else if(min(d)==0 & max(ta[n2,]>0)){
outmat[n1,n2]=2 ## Subsuming
slist[[n1]]=c(slist[[n1]],n2)
}
}
}
}
rownames(outmat)=colnames(outmat)=rownames(ta) #headers
## loop through slist to remove trivial relationships and identify equality through subsumed categories ----
if(!sub){ ## remove indirectly subsumed categories (ie. trivial relationship A -> C in A -> B -> C)
ta_agg <- matrix(0, nrow=1,ncol=dim(ta)[2]) ## setup matrix for agg rmat of each item in list
item_list = list() ## capture item_code not overlapped in a category
for(i in 1:length(slist)){
item_list[[i]]=c(NA)
so = c()
for(s in slist[[i]]){
if (!is.na(s)){ ## disregard NAs in list
so = c(so,slist[[s]])
ta_agg = ta_agg + ta[rownames(ta)[s],] ## agg rmat for each subsumed item
}
}
ta_agg[ta_agg > 1] <- 1 ## replace values > 1 in agg matrix to 1
d_s = ta[rownames(ta)[i],] - ta_agg
catsub=F
if(min(d_s)==max(d_s)){
catsub = T ## boolean variable to avoid resolving checks on each subsequent for loops
} else {
item_list[[i]] = colnames(ta)[d_s[1,] >= 1] ## retrieve residual item_codes not covered by subsumed categories
## not correct. if category is subsumed by another, then item_code is overlapped.
}
for(s in slist[[i]]){
if (!is.na(s)){
if(s %in% so){
outmat[i,s]=0
} else if(catsub) {
outmat[i,s]=3 ## Equality via aggregate of subsumed items
}
## loop to remove subsumed identical relationships
if (!is.na(ilist[[s]])) {
if (s != ilist[[s]]){
outmat[i,s] = 0
}
}
}
}
## if category is identical and not oldest/smallest category, remove all non identical relationship
if(!is.na(ilist[[i]])){
if (i != ilist[[i]]){
outmat[i,][outmat[i,] > 1] <- 0
}
}
}
}
return(outmat)
}
