I'm trying to come up with a design for an experiment. Please bear in mind that while this dataset covers medical data, I will consult a statistician before actually implementing the design if I were to decide to do so. For now, I'm starting to gather ideas for my own education.

I'm working with a dataset of patients that underwent a certain surgical operation. The outcome is improvement vs no improvement. Based on this dataset, a model was trained using logistical regression. The model was trained using data from 2014 until 2018. Since the surgical operation is a relatively new treatment, it is expected that treatment outcome would improve over the years to due increased experience and new surgical techniques. The model does not contain variables that quantify surgical quality.

My hypothesis is that the model would therefore underestimate treatment outcome for new patients, while overestimating treatment outcome for patients before 2014.

I had a few ideas in mind, please let me know if they are flawed or if there is a better way to test it.

  1. My main idea would be to use the existing trained model and test it using data from 2019+2020, as well as 2012+2013. This testing data is not included in the trained model. If my hypothesis is correct, then I would see an underestimation for positive outcome in the most recent testing set, while seeing an overestimation in the older testing set.

  2. My second idea was to gather all the data as a whole (2012 until 2020), then split it into 3: (20% testing, 60% training, 20% testing). This would be similar to 1, except it would mean that I were to construct a temporary new model. The advantage would be that I will have more data to test with and more equally distributed. I think it would possibly yield an indication that the effect of improvement over the years exist. Since my primary goal is to assess the current existing model, I don't prefer this method.

  3. My third idea would be to use the existing model, and test it on all the years it was trained with. (2015 until 2018). I would then compare the 4 included years. This would show an overestimation for early years, and an underestimation for later years. I think this is flawed because you are not supposed to use training data to test the model, however I am not sure since all my testing sets I am comparing are included.

  4. My last idea would be to simply plot a histogram of the percentage of positive treatment outcome over the years. This may show me that this effect exists, but I don't think it would be helpful in assessing the model since the model may already account for underlying confounders I am not aware of.

Are my ideas flawed in any manner I am not yet aware of? Is there a better way to test this hypothesis?

  • $\begingroup$ Is there a control group? Or is everyone treated with the same type surgery? $\endgroup$
    – dimitriy
    Mar 6 '21 at 16:34
  • $\begingroup$ Everyone is treated with the same surgery $\endgroup$
    – Laurens
    Mar 6 '21 at 18:57

The word experiment is incorrect then, since an experiment is a randomized controlled trial, but you just have a trial. In this case, a regression model for success with dummy variables for year of treatment would be appropriate, but with heteroskedasticity-robust standard errors rather than the standard OLS ones.

A logit would also work, but probably not worth the bother since the answer would be very similar to the regression if your goal is to compare the success rate across time.

This assumes that each patient is observed once, you know when he was treated, and success is defined over a period of the same post-surgery length. That is, you have success in x years after surgery, and each patient has completed x years of follow-up, so there are no data maturity issues.

If you have covariates, like age and gender that you want to adjust for, they can be added to the model.

  • $\begingroup$ Thank you for your clear response. I may have not clearly stated in my post that my outcome is dichotomous (improvement vs no improvement). Would this mean that the only option for me is logistic regression? The existing model to predict outcome is already based on logistic regression. Would then the most logical step be to simply add a dummy variable for year of treatment to this model, to then obtain the regression coefficient for year of treatment? Or am I better off to create a new logistic regression model using fewer variables? $\endgroup$
    – Laurens
    Mar 7 '21 at 13:32
  • $\begingroup$ The regression is fine with dichotomous outcomes. This is a subtle point, but you are not modeling a binary outcome, but its expected value, so the linear probability model here is not a stretch. But if you already have a logit, you can stick with that. I would fit both the simple model and the more complicated one and report average marginal effects of the year dummies. $\endgroup$
    – dimitriy
    Mar 7 '21 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.