Statistical significance of changing a hyperparameter Suppose we have a machine learning model (e.g. an SVM) and we've run a  search over multiple hyperparameters (e.g. kernel, C) to gauge performance.
How would we gauge whether changing one parameter (e.g. C: 1.0 -> 1.5) creates a statistically significant difference in performance? Could we determine aggregate statistical significance over a parameter (e.g. changing C affects model performance more than changing the kernel over all trials)?
 A: Why do you wonder whether a change is statistically significant? With machine learning approaches you would usually tend to pick the value with best prediction performance (or average over those with close to best performance or similar things - the list of possibilities is endless, but picking the best one is pretty common). One standard way of doing this evaluation is cross-validation, which you could do over lots of different choices combined (in the kind of situation you describe above). Whether something is "statistically significant" does not really matter one way or another.
When it is about anything other than choosing what parameter value to use, then I am also not sure whether hyperparameters of machine learning approaches can be meaningfully interpreted. This is in contrast to e.g. hyperparameters in some classical hierarchical models, where you may be able to interpret e.g. the hierarchical scale parameter in terms of how much the outcome varies across subjects.
If you must have some form of statistical significance, then in principle you can of course compare the performance of a combination of hyperparameter choices on a test set.
