How to augment imaging data for deep learning if I have subject meta information about the original images? I'd like to try deep learning on about 200 CTs of cancer patients and I gave a lot of meta information about these people that I'd like to make use of in the classification (age, body mass index, chemotherapy response, etc). If I start augmenting the image data set using rotations and such, how do I deal with the fact these are the same patients? I cannot really start scrambling  the clinical information just to fake a larger database. Or can I? So, how would I "augment" the clinical variables?
 A: Why can you manipulate images
The only reason that dataset augmentation works in images is that we have some previous knowledge on the structure and contents of our data. For instance, I know that a picture of an 8 is still an 8 if I rotate it 5 degrees, or if I add gaussian noise. In fact, because it is an 8, I can also flip the image horizontally or vertically!
I know these to be valid manipulations (augmentations) of this dataset because I know some prior information about the dataset.
Manipulating health data
I personally know little about patient health information and its relevance to cancer patient classification, but maybe you do know some information. Think about the BMI, is a BMI of 20 that different from a 21? Could you perturb the BMI column by adding some gaussian distributed noise to it without affecting the meaning of the data? Same with age, when you are young 2 years old is very different from 3 years old. 55 years old is not that different from 56. Can you take this into account to make perturbations as a function of age?
As a data scientist, it is your job to either be an expert on your data or find someone who is. Use that knowledge to make reasonable decisions about transforming, manipulating, or augmenting your data. Make sure to use valid statistical methods to validate your results (cross-validation) and compare existing manipulations to not manipulating, and see for yourself what works.
Why augment at all
The main point that I would like to get across is: why would someone want to augment their dataset, and what would they gain? Dataset augmentation cannot make the neural network (correctly) learn patterns that are not there. It may learn (incorrect) patterns that you introduce with the perturbations, but this is not likely what you want. Instead, we augment the data to make the network robust to noise within the dataset. So to do this we introduce the noise ourselves as a form of regularization. If the noise we introduce is structured differently from real noise, we may overfit to the wrong type of noise and hurt our network.
For instance, if the noise associated with BMI is gaussian but we apply a uniform noise, the network may fail on real data.
It sounds like your data set is quite small, and I'm guessing you want to augment it so that deep learning is possible. It seems likely that you've seen that you need thousands (millions [billions]) of samples to train a deep model, but unfortunately you cannot create data that you don't already have.
Unless you have a simulator. Then you can create data you don't have. But I don't think we have a reliable cancer patient simulator (but I probably wouldn't know even if it did exist).
Concrete suggestions
Test it empirically! The best way to know if you can do something is to do it with valid statistical testing.


*

*Don't augment your data and test on a train/test split. How well does it perform?

*Augment the data by only perturbing the image portion, copy the metadata exactly as is between augmentations. Split the data (make sure the test set has no perturbations), how well does it perform?

*Come up with a strategy for perturbing some or all of the metadata fields. Augment the data with perturbations on these and not on the images. Split the data, how well does it perform?

*Use the strategy to augment the data perturbing both the images and the metadata. Split the data. How well does it perform?

A: I think it is hard to say definitively what the effect of changing clinical characteristics would be, in contrast to image augmentation.
The idea behind image augmentation is that say rotating an image does not change the label.  You know this, but the network does not.  So it is not just making a larger dataset (you could just duplicate all images and get a larger dataset), but actually a way to provide more information to the algorithm. In the case of rotations, you say that objects can be rotated and essentially remain the same.
Scrambling the age however might change the label (risk might be a function of age).  If you are certain that changing age by, eg, only a few days has no effect on the label, you could try that---essentially adding some noise.  However, doing so probably is not adding a lot of new information.  There might be ways to scramble certain variables that would provide more information, but it is hard to say what these might be.  Rotating images and claiming invariance under rotation in contrast is a lot of new information.  
You might get best results if you train the image network separately so it does not learn to depend on clinical characteristics that are fixed over augmentations, but you also might get worse results if the images and clinical characteristics interact a lot.
Related to this last point is a new paper that proposes “cross modal” LSTMs, in which features are extracted separately from different modalities, but cross connections allow information flow between them.
See 
Cross-modal recurrent models for weight objective prediction from multimodal time-series data.  Velickovic et al. 2017. ML4H 2017 nips workshop.
