# How to handle bounded [0,1] dependent variable that causes one to fail heteroscedasticity

In my particular situation, our outcome variable is recall (bounded between 0 and 1 inclusive), and we are building a linear mixed effects model in R. We end up with a qq plot like the one below:

Is there anything to do to deal with the bounded outcome variable? Any transformations I make will also be bounded, so I don't see how to get around this. Or is the general idea to just do any transformation I can to get the residuals as close to constant variance as possible, and the shape caused by the bounds isn't a big deal?

Some similar questions:

• Why not analyze your data using the zero-one inflated beta regression mixed effects modeling using the glmmTMB package, for example? See cran.r-project.org/web/packages/glmmTMB/vignettes/glmmTMB.pdf. May 22, 2018 at 19:25
• I don't think glmmTMB can do zero-one inflation (only zero-inflation) ? But you could shrink the data a little bit inward to get (0,1) (e.g. see Smithson and Verkuilen's "better lemon-squeezer" paper) May 22, 2018 at 20:24
• It looks like you might have discrete denominators in your data (downward-sloping linear features in your residual plot); can you use a binomial model? May 22, 2018 at 20:25
• @BenBolker The denominators are discrete; the measure is recall (tp / (tp + fn)). However, each observation (a user, in this case) may have a different number in the denominator, which makes me think that a binomial model cannot be used... Does that sound right to you? (I haven't used binomial models in awhile) May 22, 2018 at 21:35
• you absolutely can use a binomial model, and that would be the right thing to do. In lme4 either cbind(tp,fn) ~ ... or tp/(tp+fn) ~ ..., weights=tp+fn) May 22, 2018 at 22:41

If you really had [0,1] data with no definable denominator, you could use a mixed model with a Beta-distributed response (e.g. in the glmmTMB or brms packages in R), but you would need to do something about the exact 0 and 1 values (which are not feasible for a Beta response - they have likelihood density of either 0 or infinity unless the shape parameters are exactly (1,1)), e.g. shift them slightly toward 0.5 (see e.g. Smithson and Verkuilen Psychological Methods 2006).
the measure is recall (tp / (tp + fn)). However, each observation (a user, in this case) may have a different number in the denominator ...
I would recommend that you use a binomial model. In lme4 (and other R packages) you can either (1) specify the response as a two-column response cbind(success,failure) ~ ... or (2) specify the proportion tp/(tp+fn) ~ ... as the response, and include a weights argument that gives the denominator (tp+fn).