The correct way to estimate epistatic variance is to model step by step your genetic terms. For example, you would have to first model the additive component and then force those variance components in the next model and then add the dominance, and so on for the epistatic term.
An example in the documentation of sommer 3.7 to do this forcing exercise:
data(DT_cpdata)
#### create the variance-covariance matrix
A <- A.mat(GT) # additive relationship matrix
#### look at the data and fit the model
head(DT)
mix1 <- mmer(Yield~1,
random=~vs(id,Gu=A)
+ Rowf + Colf,
rcov=~units,
data=DT)
summary(mix1)
####=========================================####
#### adding dominance and forcing the other VC's
####=========================================####
DT$idd <- DT$id;
A <- A.mat(GT) # additive relationship matrix
D <- D.mat(GT) # dominance relationship matrix
mm <- matrix(3,1,1);mm ## matrix to fix the var comp
mix2 <- mmer(Yield~1,
random=~vs(id, Gu=A, Gt=mix1$sigma_scaled$`u:id`, Gtc=mm)
+ vs(Rowf,Gt=mix1$sigma_scaled$Rowf, Gtc=mm)
+ vs(Colf,Gt=mix1$sigma_scaled$Colf, Gtc=mm)
+ vs(idd, Gu=D, Gtc=unsm(1)),
rcov=~vs(units,Gt=mix1$sigma_scaled$units, Gtc=mm),
data=DT)
summary(mix2)