When is it okay to make changes to your model after validating? Let’s say I’m building a model to predict cancer relapse for a scientific paper. I use my training set to build many models and validate the best one on my test set to get an AUC of 0.65. I then go back and tweak a few hyperparameters and get a validated AUC of 0.80. Have I compromised my results by using my test set to reselect the best model? What is the best way to maintain integrity in my test/train split in this situation?
I’m fairly new to data science so I'm really curious how other people approach this problem.
Edit: This is a very small dataset so test/train/validate splits aren't possible here
 A: The purpose of testing your model on data it hasn't seen (i.e the test set) is to obtain an unbiased estimate of the model's true accuracy.  It is important to note, that this is an estimate of the true accuracy, and not the true accuracy itself.
Now, if you calculate two unbiased estimators, but only report the maximum of the two, your estimator is no longer unbiased (assuming there is some overlap between the distributions of the two estimators).
So to answer your questions:

Have I compromised my results by using my test set to reselect the best model?

Yes, because your estimate of the true accuracy is no longer unbiased.

What is the best way to maintain integrity in my test/train split in
  this situation?

There are two solutions here.


*

*Use a train/test/validation split.  Essentially do exactly what you have done, but include an extra validation dataset to test your final model on.

*Use k-fold cross validation on your training set to select your model. Then calculate your accuracy estimate on the test set.


In both scenarios, the final accuracy estimate should only be calculated once, otherwise it is no longer an unbiased estimator.
A: 
When is it okay to make changes to your model after validating?

That is OK, iff:


*

*the changes are of a "benign" nature, i.e. the risk that the model gets worse rather than better is small.
One typical example for this is: after successful internal validation (verification), we know that error/accuracy fulfills the specifications with a sensible safety margin, and the training procedure leads to stable models.
In this situation, it is fine to train the model that is actually used on the full data set with the same training function/hyperparameters as the surrogate models that were actually tested.
(You may see this scenario as a spelled out variant of important assumptions behind cross validation where we routinely use the surrogate models' performance as approximation for the performance of the model whose performance we actually need)


and


*

*actual performance is not very critical for the specific application
(read: never in medical diagnostics), or

*it is clear that the internal validation is only an intermediate step, and a full validation study will follow. In that case, you may skip an additional internal validation of the actual final model. (OTOH, skipping that you my find difficulties in convincing the ethics committee and funding agency that the method is ripe for full validation study.)



Now, in small sample size in the test set is one situation where we actually have to expect large bias after selecting/tuning based on these test results. This, together with medical diagnostic being a "critical" field where one would rather go with conservative estimates about the performanes makes the idea particularly bad in your case.
If I cannot afford to split the data set in three, I fix the hyperparameters beforehand based on my experience with the modeling approach, data generation process and the application. If that isn't possible, I go for a model that allows this approach. 
Also, to get the best out of a small data set, every splitting is done by repeated k-fold cross validation (for a train/validate/test approach that would be repeated nested k-fold cross validation). 

Last but maybe not least: AUC is just one of the figures of merit you evaluate, right?
In my experience with medical diagnostics, AUC has never been a very good decision criterion or figure of merit to evaluate whether the model is fit for purpose: diagnostic applications typically have particular needs such as requiring high sensitivity or high specificity and of two classifiers with equal AUC, one may be fit for purpose and the other one totally unacceptable. 
