1
$\begingroup$

I am reading a paper which estimates the survival function with a nonparametric estimator (e.g. Kaplan-Meier) and then uses machine learning methods to estimate the complete survival function. This is compared with the "traditional two-parameter Weibull reliability estimation". The paper doesn't describe how the Weibull parameters are estimated and I couldn't find information about what is the "traditional method".

Also the error is compared with tenfold-crossvalidation. This is clear for the machine learning methods, but I I have no idea how you would do this with the traditional method. I calculated a survival function with the weibullfitter method from lifelines in python for one dataset that was provided and calculated the error with respect to the nonparametric estimation, but I got higher errors, so that's probably not what they are doing.

The paper is called "On the use of machine learning methods to predict component reliability from data-driven industrial case studies" from Alsina, Chica, Trawinski, Regattieri (doi: 10.1007/s00170-017-1039-x)

Can somebody help me with which method is used to estimate the Weibull parameters and how (or if you even can) calculate the error for the Weibull estimation? A link/book or somewthing where I can look this up would be sufficient.

$\endgroup$
0
$\begingroup$

The "traditional two-parameter Weibull" model is presumably what you got from the lifelines package. That gives you a fully parameterized functional form for survival/reliability as a function of time.

For the error estimate used in the cited paper, Section 4.2 says:

The used fitting evaluation measure is the mean squared error (MSE) which will show the reliability fitting performance of the machine learning method. MSE computes the sum of the squares of the output differences at the $i^{th}$ time unit between the value of the approximate failure distribution and the output from the machine learning models...

The "approximate failure distribution" is presumably the Kaplan-Meier (KM) survival estimate, which gives a step-function estimate of survival as a function of time. The "output from the machine learning models" is the reliability/survival estimate from each model at each evaluated time point. So you just take the average of squared differences between observed (KM) and modeled survival values over a set of time points.

At least at first reading, however, it's not clear just which time points $i$ the authors included in their evaluations. That might be why you have trouble reproducing their results with the Weibull model. They could have used each individual time value along the time span of the data set, giving an overall, effectively integrated, estimate. Or they could have restricted evaluation to observed event times. The calculated MSE would differ depending on that choice of evaluation times.

Unlike for the machine-learning approaches, which use cross validation to optimize parameters for model fit, fitting a Weibull model does not require cross validation (CV) as that's done by maximum likelihood. You could do CV or bootstrapping, however, to evaluate the quality of the modeling process. That can give a sense of how well your model is likely to work on a new data sample.

If you choose to use the MSE survival/reliability criterion for that type of Weibull modeling, you would then fit a Weibull model on each "training" set (held-in cases in CV, bootstrap sample in bootstrap) and calculate the MSE against the corresponding KM curve of the "test" set (held-out cases in CV, entire data set in bootstrap). Multiple CV runs or bootstraps will give the most robust results.

$\endgroup$
4
  • $\begingroup$ Thanks a lot for the answer! It seems that for the dataset provided (Code E) they made a mistake when calculating the Weibull MSE for complete/censored data as it doesn't coincide with the difference between the complete and censored data in the following table. With the lifelines package I got the same difference between the observations, so that value should be correct. What I am still wondering is the comparison of the metrics. They compare CV mean squared loss for machine learnings method with max. likelihood squared error loss for the whole data $\endgroup$
    – Aimbo
    Nov 15 at 14:44
  • $\begingroup$ @Aimbo my sense is that you are correct about the difference in MSE metrics; it's hard to compare the CV MSE for machine learning methods with the Weibull whole-model MSE values. It's good that you thought this through for yourself, not just taking the publication at face value. A better comparison would have been repeated model building of all types on multiple bootstrap samples of the data sets, of the type indicated in the last paragraph for Weibull models. But the authors apparently didn't do that. $\endgroup$
    – EdM
    Nov 15 at 14:51
  • $\begingroup$ Would you say bootstrapping is better than CV for estimating the quality of the modelling process in this case? $\endgroup$
    – Aimbo
    Nov 16 at 10:55
  • $\begingroup$ @Aimbo both can work in principle, but Frank Harrell estimates that "To be as good as the bootstrap, about 100 repeats of 10-fold cross-validation are required." See also this paper, recommending bootstrap over cross validation in the related context of logistic regression. $\endgroup$
    – EdM
    Nov 16 at 13:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.