Test/validate a random forest survival model

Question

I have around 7,588 features for 100 subjects with time to event and status data. I have built 100 random forest survival models with 75/25 train-test splits using ranger() in R. From the 100 training runs, I have fetched the variable importance and took the mean and standard deviation to show the top ranking variables across the 100 runs. For the test data, I can do predictions using the predict() function in R, but I don't know how to show the accuracy of those 100 models on their respective test data. I intend to get one metric per test that I can plot for all the 100 models to show that the models from which I am fetching variable importance are good enough. Any suggestion would be helpful. Thanks.

From the predicted object I get.

pred$unique.death.times
 [1]  1  2  3  4  5  6  7  8  9 10 11 12 13 14 16 18 19 20 22 24 25 28 30


pred$chf
             [,1]       [,2]       [,3]       [,4]       [,5]       [,6]      [,7]      [,8]      [,9]     [,10]     [,11]
 [1,] 0.003433333 0.02085000 0.03366667 0.06342619 0.10389286 0.12087619 0.1625262 0.1742429 0.1946595 0.2372595 0.2614929
 [2,] 0.030750000 0.16698333 0.82955833 1.00299167 1.09204167 1.14710833 1.1674012 1.2102012 1.2260679 1.2426512 1.2429012
 [3,] 0.010866667 0.05631667 0.12265714 0.16350714 0.24914048 0.28680714 0.3197238 0.3524738 0.3932905 0.4411905 0.4715071
 [4,] 0.010300000 0.05953571 0.12953571 0.20223571 0.43103333 0.96713333 1.0549167 1.1070333 1.1771833 1.2306167 1.2484000
 [5,] 0.006966667 0.03738333 0.06581667 0.11755952 0.21300952 0.26141190 0.7309286 0.8148452 0.8915452 1.0537286 1.1087952
 [6,] 0.048966667 0.55072619 0.72814286 0.86837381 0.98765714 1.04730714 1.0972833 1.1631000 1.2101500 1.2435333 1.2643000
 [7,] 0.013600000 0.07807778 0.13650278 0.21750278 0.79135278 0.94225278 1.0154194 1.0470194 1.0953528 1.1392028 1.1692361
 [8,] 0.005283333 0.02788333 0.04301667 0.06031667 0.07996667 0.09191667 0.1164000 0.1275833 0.1389333 0.1808333 0.2053833
 [9,] 0.002166667 0.02291667 0.03801667 0.07395000 0.11111667 0.14405000 0.1862024 0.1969190 0.2126357 0.2494690 0.2650357
[10,] 0.018050000 0.06731667 0.13766667 0.21378333 0.29263333 0.33341667 0.3712667 0.3958833 0.4144667 0.4353333 0.4435333
[11,] 0.003750000 0.03561667 0.06907381 0.10042381 0.14879048 0.18882381 0.2201405 0.2358905 0.2659214 0.3141214 0.3524214
[12,] 0.008233333 0.05555000 0.12008333 0.18090000 0.37871667 0.89438571 0.9985524 1.0800357 1.1519857 1.2261357 1.2570524
[13,] 0.014016667 0.09945000 0.65139048 0.86217381 1.04141548 1.10203214 1.1285821 1.1382821 1.1494488 1.1717917 1.1768750
[14,] 0.005116667 0.03768333 0.06548333 0.09837619 0.16193333 0.20111667 0.2986000 0.3542000 0.7996167 0.9953167 1.0735833
[15,] 0.046027778 0.51163611 0.72285278 0.85528611 0.98585278 1.02413611 1.0741528 1.1001361 1.1391528 1.1758694 1.1948337
[16,] 0.008700000 0.05448333 0.08956667 0.16113333 0.22355000 0.24701667 0.2786167 0.2872667 0.3089000 0.3382429 0.3579929
[17,] 0.015883333 0.12312619 0.60224405 0.76841071 0.90501071 0.95872738 1.0095274 1.0506274 1.0880940 1.1359440 1.1469440
[18,] 0.012616667 0.07705000 0.27295952 0.83005952 1.12160952 1.21302619 1.2366095 1.2625929 1.2798262 1.2949929 1.3013262
[19,] 0.026283333 0.44663333 0.55680000 0.80888333 0.92163333 0.95196667 1.0187000 1.0353167 1.0616500 1.1094000 1.1527333
[20,] 0.006933333 0.04388333 0.07006667 0.09783095 0.13756429 0.16238095 0.2367476 0.2659976 0.3138143 0.4826655 1.0050988
[21,] 0.003783333 0.02503333 0.06333333 0.08868333 0.14063333 0.17550000 0.2095333 0.2362833 0.2613333 0.3217333 0.3556667
[22,] 0.003833333 0.03439286 0.06794286 0.14052381 0.61407381 0.74754048 0.8760571 0.9338238 0.9969571 1.0961905 1.1414405
[23,] 0.039994444 0.17944722 0.37164008 0.96392341 1.17032341 1.22225675 1.2553901 1.2754734 1.2974901 1.3080734 1.3142901
          [,12]     [,13]     [,14]     [,15]     [,16]     [,17]     [,18]     [,19]     [,20]     [,21]     [,22]
 [1,] 0.4137595 0.9204262 1.1763595 1.1944762 1.2571595 1.2792095 1.3207262 1.3460595 1.3600762 1.4005595 1.4005595
 [2,] 1.2574512 1.2611179 1.2707345 1.2720679 1.2769845 1.2769845 1.2799845 1.2803179 1.2853179 1.2863179 1.2863179
 [3,] 1.0253155 1.1405155 1.2705155 1.2898488 1.3167655 1.3227321 1.3488988 1.3610655 1.3756321 1.3890321 1.3890321
 [4,] 1.3096500 1.3347833 1.3643500 1.3662667 1.3809667 1.3903833 1.4137000 1.4210333 1.4267833 1.4329500 1.4329500
 [5,] 1.1975119 1.2461619 1.3099619 1.3227286 1.3573786 1.3684119 1.3836786 1.3955952 1.4036786 1.4291286 1.4291286
 [6,] 1.3136833 1.3276833 1.3436167 1.3453167 1.3614000 1.3656500 1.3713167 1.3753167 1.3780667 1.3784000 1.3784000
 [7,] 1.2284028 1.2562194 1.2974694 1.3088861 1.3206028 1.3276861 1.3492028 1.3584361 1.3645028 1.3713361 1.3713361
 [8,] 0.2974667 0.3435667 0.4050667 0.4184833 0.5048000 0.5578333 0.7184571 1.2129571 1.3432905 1.4455238 1.4455238
 [9,] 0.3046690 0.3301190 0.3840857 0.3943357 0.4279119 0.4434619 0.4637702 0.4771536 0.5044702 0.5660702 0.5660702
[10,] 0.4815250 0.4911750 0.5200083 0.5311750 0.5437083 0.5535750 0.5749083 0.5874083 0.6191250 0.6455750 0.6455750
[11,] 0.5357857 1.0566357 1.2589357 1.2955524 1.3538024 1.3762190 1.4087119 1.4330452 1.4458286 1.4526619 1.4526619
[12,] 1.3287857 1.3494690 1.3854190 1.3907524 1.4036690 1.4123524 1.4377857 1.4434524 1.4512357 1.4559024 1.4559024
[13,] 1.2070083 1.2164250 1.2446250 1.2478750 1.2568679 1.2677345 1.2933179 1.3024845 1.3109179 1.3205179 1.3205179
[14,] 1.1977833 1.2502167 1.3286833 1.3412500 1.3749500 1.3843500 1.4058833 1.4187000 1.4329000 1.4411333 1.4411333
[15,] 1.2355671 1.2524171 1.2863004 1.2928671 1.3114337 1.3150171 1.3334837 1.3394837 1.3492671 1.3580837 1.3580837
[16,] 0.4282762 0.4931262 1.0763429 1.1478429 1.2390929 1.2567929 1.2916429 1.3076262 1.3231262 1.3463929 1.3463929
[17,] 1.2185274 1.2303607 1.2669274 1.2761107 1.2932107 1.3058940 1.3283440 1.3391107 1.3557440 1.3618274 1.3618274
[18,] 1.3309095 1.3344929 1.3480262 1.3482762 1.3526095 1.3581929 1.3647095 1.3675429 1.3714095 1.3739095 1.3739095
[19,] 1.1955976 1.2531476 1.2896810 1.2969310 1.3214976 1.3268810 1.3423643 1.3629476 1.3695643 1.3748643 1.3748643
[20,] 1.2462321 1.3727655 1.4465488 1.4585821 1.4803821 1.4878155 1.4979488 1.5071988 1.5111155 1.5189488 1.5189488
[21,] 0.9377333 1.0617500 1.1885333 1.2006167 1.2409667 1.2581000 1.2897167 1.3085833 1.3284000 1.3429000 1.3429000
[22,] 1.2062905 1.2372071 1.2867405 1.2929238 1.3290405 1.3437738 1.3594405 1.3724738 1.3866571 1.3947405 1.3947405
[23,] 1.3292067 1.3335901 1.3440067 1.3461734 1.3496734 1.3511734 1.3548401 1.3578401 1.3600401 1.3600401 1.3600401
          [,23]
 [1,] 1.4005595
 [2,] 1.2863179
 [3,] 1.3890321
 [4,] 1.4329500
 [5,] 1.4291286
 [6,] 1.3784000
 [7,] 1.3713361
 [8,] 1.4455238
 [9,] 0.5660702
[10,] 0.6455750
[11,] 1.4526619
[12,] 1.4559024
[13,] 1.3205179
[14,] 1.4411333
[15,] 1.3580837
[16,] 1.3463929
[17,] 1.3618274
[18,] 1.3739095
[19,] 1.3748643
[20,] 1.5189488
[21,] 1.3429000
[22,] 1.3947405
[23,] 1.3600401


pred$survival
           [,1]      [,2]      [,3]      [,4]      [,5]      [,6]      [,7]      [,8]      [,9]     [,10]     [,11]
 [1,] 0.9965726 0.9793659 0.9668937 0.9385434 0.9013219 0.8861437 0.8499938 0.8400929 0.8231149 0.7887866 0.7699014
 [2,] 0.9697180 0.8462137 0.4362419 0.3667805 0.3355308 0.3175537 0.3111746 0.2981373 0.2934442 0.2886180 0.2885459
 [3,] 0.9891922 0.9452398 0.8845669 0.8491604 0.7794705 0.7506565 0.7263496 0.7029470 0.6748327 0.6432702 0.6240610
 [4,] 0.9897529 0.9422019 0.8785032 0.8169023 0.6498372 0.3801713 0.3482214 0.3305381 0.3081455 0.2921124 0.2869636
 [5,] 0.9930575 0.9633068 0.9363025 0.8890876 0.8081484 0.7699637 0.4814617 0.4427078 0.4100217 0.3486354 0.3299562
 [6,] 0.9522129 0.5765310 0.4828048 0.4196334 0.3724483 0.3508814 0.3337766 0.3125159 0.2981526 0.2883635 0.2824369
 [7,] 0.9864921 0.9248925 0.8724039 0.8045254 0.4532313 0.3897488 0.3622505 0.3509823 0.3344216 0.3200741 0.3106041
 [8,] 0.9947306 0.9725018 0.9578954 0.9414664 0.9231471 0.9121812 0.8901191 0.8802201 0.8702860 0.8345744 0.8143351
 [9,] 0.9978357 0.9773439 0.9626969 0.9287181 0.8948343 0.8658445 0.8301056 0.8212571 0.8084506 0.7792144 0.7671786
[10,] 0.9821119 0.9348991 0.8713891 0.8075233 0.7462957 0.7164716 0.6898600 0.6730852 0.6606926 0.6470489 0.6417648
[11,] 0.9962570 0.9650101 0.9332578 0.9044540 0.8617497 0.8279324 0.8024061 0.7898672 0.7664994 0.7304303 0.7029838
[12,] 0.9918005 0.9459647 0.8868465 0.8345188 0.6847396 0.4088587 0.3684124 0.3395834 0.3160086 0.2934243 0.2844914
[13,] 0.9860811 0.9053352 0.5213204 0.4222432 0.3529547 0.3321953 0.3234916 0.3203689 0.3168113 0.3098114 0.3082405
[14,] 0.9948964 0.9630178 0.9366147 0.9063079 0.8504979 0.8178170 0.7418561 0.7017346 0.4495012 0.3696064 0.3417816
[15,] 0.9550154 0.5995139 0.4853656 0.4251615 0.3731209 0.3591066 0.3415870 0.3328258 0.3200901 0.3085506 0.3027543
[16,] 0.9913377 0.9469743 0.9143273 0.8511786 0.7996749 0.7811277 0.7568300 0.7503116 0.7342542 0.7130221 0.6990781
[17,] 0.9842421 0.8841521 0.5475815 0.4637495 0.4045376 0.3833805 0.3643912 0.3497183 0.3368579 0.3211188 0.3176059
[18,] 0.9874626 0.9258436 0.7611236 0.4360233 0.3257551 0.2972962 0.2903670 0.2829195 0.2780856 0.2738998 0.2721706
[19,] 0.9740591 0.6397785 0.5730399 0.4453551 0.3978687 0.3859812 0.3610640 0.3551139 0.3458846 0.3297568 0.3157725
[20,] 0.9930906 0.9570656 0.9323317 0.9068022 0.8714783 0.8501173 0.7891904 0.7664410 0.7306547 0.6171362 0.3660085
[21,] 0.9962238 0.9752774 0.9386305 0.9151353 0.8688078 0.8390374 0.8109626 0.7895569 0.7700242 0.7248915 0.7007062
[22,] 0.9961740 0.9661919 0.9343139 0.8689030 0.5411419 0.4735298 0.4164216 0.3930479 0.3690006 0.3341416 0.3193587
[23,] 0.9607948 0.8357321 0.6896024 0.3813936 0.3102666 0.2945647 0.2849647 0.2792987 0.2732167 0.2703404 0.2686650
          [,12]     [,13]     [,14]     [,15]     [,16]     [,17]     [,18]     [,19]     [,20]     [,21]     [,22]
 [1,] 0.6611599 0.3983492 0.3083994 0.3028626 0.2844609 0.2782572 0.2669414 0.2602638 0.2566412 0.2464590 0.2464590
 [2,] 0.2843779 0.2833371 0.2806254 0.2802515 0.2788770 0.2788770 0.2780416 0.2779489 0.2765627 0.2762862 0.2762862
 [3,] 0.3586833 0.3196542 0.2806869 0.2753124 0.2680008 0.2664064 0.2595259 0.2563875 0.2526798 0.2493165 0.2493165
 [4,] 0.2699145 0.2632152 0.2555467 0.2550574 0.2513355 0.2489798 0.2432416 0.2414644 0.2400799 0.2386040 0.2386040
 [5,] 0.3019445 0.2876065 0.2698303 0.2664074 0.2573345 0.2545108 0.2506548 0.2476856 0.2456915 0.2395176 0.2395176
 [6,] 0.2688280 0.2650907 0.2609004 0.2604572 0.2563017 0.2552147 0.2537726 0.2527595 0.2520654 0.2519814 0.2519814
 [7,] 0.2927598 0.2847284 0.2732223 0.2701208 0.2669743 0.2650899 0.2594470 0.2570625 0.2555077 0.2537677 0.2537677
 [8,] 0.7426973 0.7092362 0.6669323 0.6580441 0.6036263 0.5724480 0.4875038 0.2973168 0.2609855 0.2356226 0.2356226
 [9,] 0.7373674 0.7188382 0.6810730 0.6741277 0.6518688 0.6418107 0.6289080 0.6205472 0.6038254 0.5677522 0.5677522
[10,] 0.6178405 0.6119070 0.5945156 0.5879138 0.5805912 0.5748909 0.5627565 0.5557658 0.5384153 0.5243609 0.5243609
[11,] 0.5852093 0.3476233 0.2839561 0.2737466 0.2582564 0.2525316 0.2444580 0.2385813 0.2355508 0.2339467 0.2339467
[12,] 0.2647986 0.2593779 0.2502189 0.2488880 0.2456938 0.2435696 0.2374530 0.2361112 0.2342806 0.2331898 0.2331898
[13,] 0.2990907 0.2962875 0.2880489 0.2871143 0.2845439 0.2814686 0.2743590 0.2718555 0.2695725 0.2669970 0.2669970
[14,] 0.3018626 0.2864427 0.2648257 0.2615186 0.2528522 0.2504866 0.2451504 0.2420284 0.2386159 0.2366594 0.2366594
[15,] 0.2906699 0.2858131 0.2762911 0.2744827 0.2694335 0.2684697 0.2635575 0.2619809 0.2594303 0.2571531 0.2571531
[16,] 0.6516314 0.6107142 0.3408397 0.3173205 0.2896468 0.2845652 0.2748189 0.2704613 0.2663015 0.2601771 0.2601771
[17,] 0.2956652 0.2921872 0.2816958 0.2791208 0.2743884 0.2709302 0.2649156 0.2620786 0.2577554 0.2561922 0.2561922
[18,] 0.2642368 0.2632917 0.2597525 0.2596875 0.2585646 0.2571250 0.2554549 0.2547321 0.2537490 0.2531155 0.2531155
[19,] 0.3025231 0.2856044 0.2753586 0.2733695 0.2667355 0.2653035 0.2612273 0.2559054 0.2542177 0.2528739 0.2528739
[20,] 0.2875863 0.2534052 0.2353812 0.2325658 0.2275507 0.2258655 0.2235883 0.2215297 0.2206637 0.2189419 0.2189419
[21,] 0.3915143 0.3458500 0.3046678 0.3010085 0.2891046 0.2841935 0.2753488 0.2702026 0.2649008 0.2610874 0.2610874
[22,] 0.2993055 0.2901936 0.2761695 0.2744671 0.2647312 0.2608594 0.2568044 0.2534791 0.2499093 0.2478974 0.2478974
[23,] 0.2646871 0.2635295 0.2607986 0.2602342 0.2593249 0.2589362 0.2579885 0.2572157 0.2566505 0.2566505 0.2566505
          [,23]
 [1,] 0.2464590
 [2,] 0.2762862
 [3,] 0.2493165
 [4,] 0.2386040
 [5,] 0.2395176
 [6,] 0.2519814
 [7,] 0.2537677
 [8,] 0.2356226
 [9,] 0.5677522
[10,] 0.5243609
[11,] 0.2339467
[12,] 0.2331898
[13,] 0.2669970
[14,] 0.2366594
[15,] 0.2571531
[16,] 0.2601771
[17,] 0.2561922
[18,] 0.2531155
[19,] 0.2528739
[20,] 0.2189419
[21,] 0.2610874
[22,] 0.2478974
[23,] 0.2566505

These are the values for (time, status) of the test data.

  AVAL CNSRFLIP
1    13        1
2     3        1
3    12        1
4     6        1
5     7        1
6     2        1
7     5        1
8    22        1
9    28        0
10   28        0
11   13        1
12    6        1
13    3        1
14    9        1
15    2        1
16   14        1
17    3        1
18    4        1
19    2        1
20   11        1
21   12        1
22    5        1
23    4        1

Answer 1

I intend to get one metric per test that I can plot for all the 100 models to show that the models from which I am fetching variable importance are good enough.

Getting a single metric per test will require some thought and care, as illustrated in the following excerpt from your predicted survival curves, the 8th and 10th rows of your predicted survival probabilities plotted against the corresponding survival times (unique.death.times):

The predicted survival curves cross. Case 8 has higher predicted survival probability than Case 10 until about Time = 19, where the order of survival probability switches. At what time point, or over which range of time points, do you want to do your comparisons of "accuracy"? How will you take censoring into account? This is a bigger problem for random forest than for Cox models, in which predicted survival curves don't cross as a function of covariates and you can, for example, simply compare the rank-order of linear-predictor values to that of survival times for all comparable pairs of cases to get a time-independent C-index/AUC.

The R pec package was designed to help with evaluating predictions from random survival forests, as explained here. It produces "prediction error curves" based on the integrated Brier score, the mean-square difference between observed survival (0/1) and predicted survival probabilities over time, while taking censoring into account. You will have to use your understanding of the subject matter to decide what time span to evaluate.

The R Survival task view section on "Predictions and Prediction Performance" has links to other approaches to evaluating time-dependent survival predictions that you might find helpful.

A further suggestion: with so few test cases, you won't have a lot of precision in your estimates and the estimates might depend highly on your test/train split. You might consider fitting the entire data set and evaluating out-of-bag performance instead.