So from what I have been able to gather I don't think there is any way to compare the models in question but I figured I would put it to better minds than my own before I gave up. Here is the situation:
The data I have follows a sigmoidal dose-response curve something like this:
df <- data.frame(Effect = c(0,0,0.025,0.05,0.1,0.15,0.2,0.3,0.4,0.55,0.7,0.8,0.9,0.95,0.97,0.98,0.99,1,1), Dose= seq(0:18)) df$Effected <- df$Effect * 100 df$Uneffected <- 100 - df$Effect plot(Effect ~ Dose, data = df)
From these data I want to determine the dose required to achieve a pre-determined effect. The effected and uneffected columns are the counts of individuals who have been effected by the dose. Classically, this has been done by fitting glms with binomial errors like so:
fit <- glm(cbind(Effected, Uneffected) ~ Dose, family = binomial(link = logit), data = df)
From here Fieller's formula is employed to back calculate the dosage required for a certain effected rate, along with confidence intervals. This formula though, as it currently stands, is incapable of dealing with quadratics meaning that we end up fitting straight lines to curved data. To make matters worse, these kinds of data are regularly over-dispersed relative to binomial dispersion - in some cases as much as 40 times the normal binomial dispersion so it is obviously an issue. There has been some consideration of using beta-binomial models to deal with this but that is something of a separate issue.
So as an alternative myself and some of my colleagues have been exploring simply swapping the terms around and modelling things like so:
fit2 <- lm(log(Dose) ~ Effect + I(Effect^2), data = df)
This type of model gets around the use of fiellers formula as we can directly predict the variable of interest and as such can use polynomials. Is this appropriate though given that dose is technically independent and effect is dependent on the response?
The other issue we have come up against is how one can determine which approach is better in a particular case. Standard metrics aren't an option as the response changes. Could one compare assumption checks? If so how? We have also considered simulation experiments and seeing which type gets consistently closest to the true value. Are there any other alternatives?
In our case, the end goal is to find the best predicting model that will give the most accurate estimates, in particular for the upper end of the effect data (i.e. > 0.9). For a given response variable we have been comparing models using cross-validation to calculate prediction error metrics (I.e. MdAE). Is there any way of doing something similar when the response variable changes but the data included in the model altogether remains the same?
Lastly, solutions would ideally be possible in R as this is the system I am most proficient in; thank you in advance!
EDIT: Fixed mistake in the glm code