I am trying to produce a linear mixed model the R code is as follows.
The response term Average.payoff is continuous whilst all explanatory variables are all binary.
When I come to validation I can clearly see that the spread of the residuals decreases with larger fitted values. Although there seems to be alot of information on heterogeneity in the form where the residuals increase with larger fitted values I have read nothing about cases similar to my own.
I have plotted the residuals against each explanatory effect and can see that the spread decreases with larger fitted values for the variables Game and Type but increases for the variable Others.Type.
What is the cause of this and what should I do about it?
Should I look at adding quadratic terms or using additive modelling? Is there a transformation that should be applied?