In addition to the excellent points in Camille's and EdM's answers:
Purpose of the model/study
Whether something sensible can be done or not depends crucially on the purpose of the modeling.
For any kind of real-world application, the random uncertainty on your internal validation results (even with the best resampling) will likely not be useful.
If on the other hand, this is a preliminary study with the purpose of generating a first rough idea to substantiate a grant proposal, that would be perfectly fine.
Random uncertainty in internal validation results
Do a back-of-the-envelope assessment of the situation.
The best resampling cannot get around the fact that you have only 60 cases (forget about hold-out), and only 20 in the smaller class.
Proportions like acurracy, sensitivity, specificity etc. have many disadvantages over the already mentioned proper scoring rules. Right here, however, they have the one advantage that they allow back-of-the-envelope calculations before even preliminary data is available.
(Proper scoring rules can have lower variance than these proportions, but don't expect miracles. Plus, overfitting leading to overconfidence in the predictions, thus predicted probabilities too close to 0 or 1, increases variance uncertainty)
With 20 patients in the negative class, specificity (the probability of correctly recognizing negative class/non-responders) will have 95 % confidence intervals (calculated via Bayes method with uniform prior):
If this uncertainty is (unjustifiably) too large for your purpose, then don't even start experiments now. Instead, you can use such confidence interval estimations to calculate a more sensible number of cases and recruit them.
Note that this includes only random uncertainty that stems from the limited number of cases. Resampling in small-sample-size situations or across optimization processes there will be additional variance due to model instability.
As I said, (strictly) proper scoring rules feature lower variance among their advantages over proportions. However, you may find that you are forced to communicate final internal validation results in the form of proportions, and then you're back to these large confidence intervals.
Looking at the variance of a figure of merit between cross validation folds gives you a funny mix of $\frac{n}{k}$ case-to-case variance and model instability variance.
(avoiding) Data-driven model optimization
These confidence intervals should also give you an idea of how large the observed difference in performance between candidate models should be to allow a data-driven choice. (Remember, choosing between multiple models -> multiple comparisons). The likely conclusion is that you cannot do data-driven model optimization with so few cases. Not even if you switch to a nested resampling strategy, and while AIC, BIC etc. spare you the inner resampling, they also cannot do miracles and will basically tell you that you cannot afford a complex model based on these case numbers. However, they may be more convenient for formulating a decision strategy for the model complexity that is then evaluated in the resampling.
Other than that, your best bet is to include as much external knowledge as possible and reduce the dimensionality of your data before even looking at it. I.e., avoid the data-driven optimization and replace it by decisions you take.
E.g. if you know you have 3 clinical parameters that are highly correlated, either decide for now to go with one (i.e. decide which one) or if you prefer with their average or sum (or difference for negative correlation) and then swallow the bill for this professional decision. The cost (in terms of performance) of being undecided will likely be heavier, because data-driven optimization costs sample size...
Or decide to go for the LASSO model using 5 parameters, without looking whether 4 or 6 look better.
The random forest you mention takes yet another strategy, and may be your best bet if you do not have sufficient external knowledge. The rF tries to "average out" model instability. The beauty here is that default parameters often yield a sensible model, so don't even think of optimizing them. If you want to be on the safe side, use more trees though: there you trade of computational resources for a possible gain in predictive performance, but you do not risk worse predictive performance by using more trees.
Leave-one-out
... is almost never advisable since it conflates surrogate model and test case, depriving you of the possibility to check stability of the predictions (there are more disadvantages to LOO, but that alone is sufficient for me to avoid it).
Instead, go for a resampling scheme like bootstrap or repeated k-fold cross validation (say, 50x10-fold or 25x20-fold) that allows you to measure instability, possibly stratified.
Is it permissible or even advisable to over-represent the smaller class (non-responders) in the test set?
Adjusting relative class frequencies is IMHO advisable only if you correct your data to better represent the application-scenario specific relative class frequencies.
All other adjustments, in particular towards better balance in the training data may lead to nice-looking results at the first glance, but that will usually come at the cost of worse performance under application working conditions.
Instead of adjusting relative class frequencies in the test data, it is IMHO better to adjust the resulting figures of merit to application-relevant relative frequencies.
Finally, be happy: I often have to work with smaller sample size and at the same time much higher dimensionality (p in the 100s or 1000s). (But I can typically use repeated measurements and external knowledge about the physics and chemistry behind the data - which makes modelling possible - even if it does not help with the uncertainty of the internal validation)
