How to handle bounded [0,1] dependent variable that causes one to fail heteroscedasticity In my particular situation, our outcome variable is recall (bounded between 0 and 1 inclusive), and we are building a linear mixed effects model in R. We end up with a qq plot like the one below:

Is there anything to do to deal with the bounded outcome variable? Any transformations I make will also be bounded, so I don't see how to get around this. Or is the general idea to just do any transformation I can to get the residuals as close to constant variance as possible, and the shape caused by the bounds isn't a big deal?
Some similar questions:


*

*Effect of baseline on change over time in mixed models?

*Continuous dependent variable with upper and lower bounds: logit transformation appropriate?

*Using t-test/ANOVA on bounded data
 A: If you really had [0,1] data with no definable denominator, you could use a mixed model with a Beta-distributed response (e.g. in the glmmTMB or brms packages in R), but you would need to do something about the exact 0 and 1 values (which are not feasible for a Beta response - they have likelihood density of either 0 or infinity unless the shape parameters are exactly (1,1)), e.g. shift them slightly toward 0.5 (see e.g. Smithson and Verkuilen Psychological Methods 2006).
However, based on the appearance of your residual plot (discrete linear features) and the comment

the measure is recall (tp / (tp + fn)). However, each observation (a user, in this case) may have a different number in the denominator ...

I would recommend that you use a binomial model. In lme4 (and other R packages) you can either (1) specify the response as a two-column response cbind(success,failure) ~ ... or (2) specify the proportion tp/(tp+fn) ~ ... as the response, and include a weights argument that gives the denominator (tp+fn).
