Survival analysis - dealing with highly censored data with computationally expensive covariates

I have approximately 1000 run-life examples (time to fail data) for equipment. However, the number of failures is quite low relative to the number of units that are censored (95% are right - censored, majority are unobserved failures, a small proportion are failures from a competing risk).

I am interested in some time-varying covariates that are quite computationally expensive to calculate for the run-life length. I am doing this retrospectively with sensor data. To reduce the computational effort, it was proposed that we could downsample those with right censoring, however this to me would bias the hazard calculation (#no. of units working / total units available). Am I right, or is this an acceptable practice under certain assumptions? And what other choices do I have with such highly censored data.

Thanks.

I don't understand how the experiment is modeling failure, but you consider failure due to "other causes" a competing risk? Are you actually modeling failure due to a specific cause?

The issue is probably caused by trying to input continuous, or semicontinuous exposures as time varying covariates. It is impossible and unnecessary for Cox models to take, as regressors, continuously varying time varying covariates. A close approximation is what most software is likely to give: wherein each registered change in a covariate value results in censoring at risk observations and re-entering them into the analysis set with the new value of the observation.

Theoretically, the same result is achieved by only updating the current value of the covariates for all observations at the time of failure for each failure in the study. So for instance, in a study of 10 where all fail, there would be 10 observations for time 0 where all take covariate values immediately preceding the first failure, 9 observations for time 1 where all take covariate values immediately preceeding the second failure, so on and so forth for 55 observations. Note: Cox doesn't care about the actual time of failure, so we can conveniently assign the risk set index (i.e. the failure's specific rank) in place of actual time. In general, you have at most (n+1)n/2 observations by inputting time varying covariates, where $$n$$ is the number of failures. While it's $$O(n^2)$$, 1,000 observations with 95% right censored is hardly a problem on modern software.

• Thanks for your reply. It is time to fail data, however we are interested in a particular failure cause, and the covariate of interest is suspected to cause that type of failure. The computational cost is not in the running of the model fit, but in the calculation of the covariate.
– Meep
Apr 23, 2020 at 22:39
• The word "cause" is being used with a lack of precision. The subset of all-cause failures that are "failures due to X" were only determined to be such after the failure happened and so if you adjust for "X" in that model, you will induce spurious associations. For instance, if I said deaths among blue eyed people were "blue eye deaths", and modeled blue eyes an exposure for such deaths, the association is immense. May 1, 2020 at 13:04

My fantastic PhD supervisor referred me to this paper (https://www.jstor.org/stable/2337580?read-now=1&seq=1#metadata_info_tab_contents) — which says:

• removing ‘incomplete’ observations increases variance in estimators (i.e we are ignoring information) — pvalues change
• more importantly to my situation, when the observations with missing covariates all have the same outcome (death/censoring) removing them introduces bias
• paper above suggests the use of an estimator to handle this particular situation.

Now I need to work out if the methodology in the paper above is implemented in any R or python packages. Happy for anyone to elaborate on my answer (explain the problem and the methodology more).