Your colleagues are incorrect.
It's possible that they're basing their suggestion on the fact that the robustness of typical parametric tests, like a t-test, to violations of assumptions is severely compromised with imbalanced N's. But if you don't violate them then a higher N is better, even if unbalanced. And you can always examine the effect of any violations through simulation.
An easy way to imagine why sampling 8 of the 100 control subjects is a bad idea, is to picture two graphs of your data that differ by the control group. Each has the patient and control brain volume measures and a confidence interval (CI) around each measure. In each graph the CI around the test group is going to be the same. But the CI around the control group will be much smaller with the full set of 100 than it will be around a sample of 8. Which graph would you like to present?
Other questions to ask yourself when deciding whether to use all 100 control samples or only 8 include: Which do you think has a better estimate of the control mean? Which would have a better estimate of the control variance? Which is more representative of the population? The answer to all of these questions is the larger group. And relating accurate estimates of your parameters is substantially more important than issues you may have with a particular test.
NOTE: It's remotely possible that the control sample of 8 CI could be smaller than the one for the full control group. However, that would likely mean that your control sample of 8 is a terrible sample and further highlights why you want to stick with the full data set.