But varying the threshold will change the predicted classifications. Does this mean the threshold is a hyperparameter?
Yup, it does, sorta. It's a hyperparameter of you decision rule, but not the underlying regression.
If so, why is it (for example) not possible to easily search over a grid of thresholds using scikit-learn's GridSearchCV method (as you would do for the regularisation parameter C).
This is a design error in sklearn. The best practice for most classification scenarios is to fit the underlying model (which predicts probabilities) using some measure of the quality of these probabilities (like the log-loss in a logistic regression). Afterwards, a decision threshold on these probabilities should be tuned to optimize some business objective of your classification rule. The library should make it easy to optimize the decision threshold based on some measure of quality, but I don't believe it does that well.
I think this is one of the places sklearn got it wrong. The library includes a method,
predict, on all classification models that thresholds at
0.5. This method is useless, and I strongly advocate for not ever invoking it. It's unfortunate that sklearn is not encouraging a better workflow.