Running a Cox regression on image features. Getting very large beta for some features. Is this valid?

Question

I have a cohort of 534 cancer patients. I have pathology slides for each patient and used a convolutional neural network trained to predict survival to extract the relevant image features. I split the data into 40/30/30 where 40% of data was used to train the cnn. I then used the cnn to extract features from the validation set.

I ran PCA on these features to reduce the number of features from 2048 to 50. I took these 50 image features and combined them with other features like age, stage, etc.

With the combined image/clinical features I ran a Cox regression and got the following result:

Iteration 1: norm_delta = 2.04734, step_size = 0.9500, ll = -531.23717, newton_decrement = 74.15919, seconds_since_start = 0.0
Iteration 2: norm_delta = 1.03262, step_size = 0.9500, ll = -479.57881, newton_decrement = 19.79851, seconds_since_start = 0.0
Iteration 3: norm_delta = 0.16658, step_size = 0.9500, ll = -460.93671, newton_decrement = 0.97542, seconds_since_start = 0.1
Iteration 4: norm_delta = 0.01867, step_size = 1.0000, ll = -459.95237, newton_decrement = 0.02205, seconds_since_start = 0.1
Iteration 5: norm_delta = 0.00068, step_size = 1.0000, ll = -459.92980, newton_decrement = 0.00004, seconds_since_start = 0.1
Iteration 6: norm_delta = 0.00000, step_size = 1.0000, ll = -459.92977, newton_decrement = 0.00000, seconds_since_start = 0.1
Convergence success after 6 iterations.

<lifelines.CoxPHFitter: fitted with 144 total observations, 14 right-censored observations>


model   lifelines.CoxPHFitter
duration col    'DxToFollowup'
event col   'IsDead'
number of observations  144
number of events observed   130
partial log-likelihood  -459.93
time fit was run    2020-01-31 21:44:51 UTC
    coef    exp(coef)   se(coef)    coef lower 95%  coef upper 95%  exp(coef) lower 95%     exp(coef) upper 95%     z   p   -log2(p)
0   0.01    1.01    0.04    -0.08   0.09    0.92    1.10    0.18    0.85    0.23
1   0.35    1.41    0.19    -0.03   0.72    0.97    2.06    1.81    0.07    3.84
2   0.78    2.18    0.38    0.02    1.53    1.02    4.63    2.02    0.04    4.54
3   -2.03   0.13    0.57    -3.15   -0.90   0.04    0.41    -3.54   <0.005  11.26
4   -2.72   0.07    0.78    -4.25   -1.18   0.01    0.31    -3.47   <0.005  10.92
5   0.85    2.33    0.73    -0.59   2.28    0.56    9.76    1.16    0.25    2.01
6   -0.22   0.81    0.90    -1.98   1.55    0.14    4.70    -0.24   0.81    0.30
7   -3.20   0.04    1.40    -5.95   -0.45   0.00    0.64    -2.28   0.02    5.48
8   -0.70   0.50    1.78    -4.19   2.79    0.02    16.22   -0.39   0.69    0.53
9   -2.64   0.07    2.32    -7.18   1.90    0.00    6.66    -1.14   0.25    1.98
10  -5.38   0.00    2.55    -10.37  -0.39   0.00    0.68    -2.11   0.03    4.85
11  3.91    49.66   2.50    -1.00   8.81    0.37    6686.40     1.56    0.12    3.08
12  -0.68   0.51    3.45    -7.45   6.08    0.00    437.37  -0.20   0.84    0.25
13  -0.08   0.92    3.33    -6.62   6.45    0.00    633.80  -0.03   0.98    0.03
14  4.96    143.27  3.15    -1.21   11.14   0.30    68668.72    1.58    0.11    3.12
15  -7.54   0.00    3.96    -15.30  0.22    0.00    1.24    -1.91   0.06    4.14
16  -4.70   0.01    4.30    -13.13  3.73    0.00    41.49   -1.09   0.27    1.87
17  2.37    10.73   4.76    -6.95   11.70   0.00    1.20e+05    0.50    0.62    0.69
18  -5.27   0.01    6.16    -17.35  6.81    0.00    906.35  -0.86   0.39    1.35
19  -7.22   0.00    6.70    -20.35  5.92    0.00    371.16  -1.08   0.28    1.83
20  19.11   1.99e+08    6.29    6.79    31.44   884.72  4.49e+13    3.04    <0.005  8.72
21  -9.61   0.00    9.12    -27.47  8.26    0.00    3879.96     -1.05   0.29    1.78
22  15.35   4.64e+06    7.77    0.12    30.59   1.12    1.92e+13    1.97    0.05    4.37
23  8.83    6865.94     7.73    -6.31   23.98   0.00    2.60e+10    1.14    0.25    1.98
24  -9.27   0.00    8.17    -25.28  6.75    0.00    854.69  -1.13   0.26    1.96
25  16.93   2.26e+07    8.74    -0.19   34.06   0.83    6.19e+14    1.94    0.05    4.25
26  6.04    421.08  10.95   -15.43  27.51   0.00    8.88e+11    0.55    0.58    0.78
27  -7.47   0.00    11.20   -29.43  14.49   0.00    1.96e+06    -0.67   0.50    0.99
28  -14.61  0.00    11.54   -37.23  8.00    0.00    2995.54     -1.27   0.21    2.28
29  -40.43  0.00    12.53   -64.98  -15.87  0.00    0.00    -3.23   <0.005  9.64
30  -18.16  0.00    11.49   -40.67  4.36    0.00    78.38   -1.58   0.11    3.13
31  -16.82  0.00    12.25   -40.84  7.19    0.00    1324.18     -1.37   0.17    2.56
32  23.38   1.42e+10    12.96   -2.02   48.78   0.13    1.53e+21    1.80    0.07    3.81
33  50.72   1.06e+22    15.60   20.15   81.29   5.63e+08    2.01e+35    3.25    <0.005  9.77
34  -22.10  0.00    14.91   -51.32  7.12    0.00    1230.88     -1.48   0.14    2.86
35  23.54   1.67e+10    15.50   -6.85   53.92   0.00    2.62e+23    1.52    0.13    2.96
36  -20.81  0.00    15.58   -51.36  9.73    0.00    16860.15    -1.34   0.18    2.46
37  -40.24  0.00    17.60   -74.74  -5.74   0.00    0.00    -2.29   0.02    5.49
38  79.00   2.03e+34    20.10   39.60   118.39  1.59e+17    2.60e+51    3.93    <0.005  13.53
39  -48.36  0.00    18.52   -84.66  -12.06  0.00    0.00    -2.61   0.01    6.79
40  20.30   6.52e+08    21.48   -21.80  62.39   0.00    1.24e+27    0.95    0.34    1.54
41  10.04   22970.68    18.67   -26.55  46.63   0.00    1.79e+20    0.54    0.59    0.76
42  26.46   3.09e+11    20.86   -14.43  67.34   0.00    1.76e+29    1.27    0.20    2.29
43  4.78    118.75  20.11   -34.64  44.19   0.00    1.56e+19    0.24    0.81    0.30
44  7.00    1099.62     23.79   -39.63  53.63   0.00    1.96e+23    0.29    0.77    0.38
45  38.28   4.20e+16    26.65   -13.95  90.50   0.00    2.02e+39    1.44    0.15    2.73
46  44.62   2.38e+19    27.66   -9.60   98.83   0.00    8.37e+42    1.61    0.11    3.23
47  26.87   4.69e+11    23.87   -19.92  73.67   0.00    9.86e+31    1.13    0.26    1.94
48  -13.06  0.00    28.91   -69.73  43.61   0.00    8.70e+18    -0.45   0.65    0.62
49  46.66   1.83e+20    33.48   -18.95  112.27  0.00    5.73e+48    1.39    0.16    2.61
AgeAtDx     0.06    1.06    0.02    0.03    0.09    1.03    1.10    3.50    <0.005  11.06
Stage_I-IB  -0.98   0.38    0.74    -2.43   0.46    0.09    1.59    -1.33   0.18    2.45
Stage_II-IIIA   -2.26   0.10    0.62    -3.47   -1.06   0.03    0.35    -3.67   <0.005  12.03
Stage_IIIB-IIIC     -2.14   0.12    0.79    -3.69   -0.59   0.03    0.55    -2.71   0.01    7.20
Stage_IV-IVC    -0.34   0.71    0.56    -1.45   0.76    0.24    2.14    -0.61   0.54    0.88
Radiation_No    1.74    5.72    0.60    0.57    2.92    1.77    18.48   2.91    <0.005  8.13
Radiation_Yes   2.08    8.00    0.66    0.78    3.38    2.17    29.43   3.13    <0.005  9.15
TumorSize   0.02    1.02    0.01    0.01    0.04    1.01    1.04    2.70    0.01    7.18
Concordance     0.78
Log-likelihood ratio test   142.61 on 58 df, -log2(p)=27.75

Several of the image features have extremely high coefficients. I am new to survival analysis and I don't know if this is something that can occur in proper models. Also, I am not sure if I should be using the training and validation set together to train the cox model, or if just using the validation set is ok.

Answer 1

My guess is that these results represent significant overfitting, with too many candidate predictors for the size of your data sample.

In the data for your table there are 130 events. The usual rule of thumb for Cox regressions on clinical data is that (without penalization) you can evaluate 1 candidate predictor per 10-20 events. One might thus argue that you could handle about 10 candidate predictors. You are trying to evaluate 58 by my count. You are stretching your data too far. I suspect that overfitting is leading to the extreme Cox regression coefficients.

One thing that could help would be to abandon the train/validation/test breakdown and use all of the data together to build the model. That breakdown typically is useful in much larger-scale studies with many thousands of cases. With this size of study you lose a lot of power and precision by breaking the data into separate groups. You are better off using all the data to build the model, then test the model-building process by repeating the entire process on multiple bootstrap samples from the data, evaluating those models on the full original data set. With all of your data you might have about 300 events, enough to evaluate about 20 predictors. As you are evaluating 8 clinical covariates that will limit you to about a dozen principal components of the image features.

You could perhaps cut down on the clinical covariates a bit to include more image components. For example, it's not clear why you have both a Radiation_No and a Radiation_Yes predictor; what's the reference value for the Radiation categorical predictor? Instead of using 4 different levels for Stage you could perhaps combine into 2. Yet there's a similar problem with the Stage data: why do you have 4 levels of Stage in your output when the listed Stage levels cover all the usual AJCC Stages; again, what's the reference level?. (My fear is that you are using "Not available" levels of Radiation and Stage as reference levels, which is not at all appropriate. That might explain the negative coefficients for the Stage coefficients, whereas higher Stage is typically associated with worse survival/positive coefficients.)

Alternatively, you could penalize the clinical covariates with ridge regression, as proposed in this study. Or you could approach the whole modeling process with LASSO, ridge, or elastic net penalization. Ridge regression is very similar to principal-components regression, except that is places relative weightings on the principal components, based on their relationships to the outcome, instead of all-or-none incorporation.