What is the "classical two parameter Weibull distribution" and cross valdiation for parametric estimation

Question

I am reading a paper which estimates the survival function with a nonparametric estimator (e.g. Kaplan-Meier) and then uses machine learning methods to estimate the complete survival function. This is compared with the "traditional two-parameter Weibull reliability estimation". The paper doesn't describe how the Weibull parameters are estimated and I couldn't find information about what is the "traditional method".

Also the error is compared with tenfold-crossvalidation. This is clear for the machine learning methods, but I I have no idea how you would do this with the traditional method. I calculated a survival function with the weibullfitter method from lifelines in python for one dataset that was provided and calculated the error with respect to the nonparametric estimation, but I got higher errors, so that's probably not what they are doing.

The paper is called "On the use of machine learning methods to predict component reliability from data-driven industrial case studies" from Alsina, Chica, Trawinski, Regattieri (doi: 10.1007/s00170-017-1039-x)

Can somebody help me with which method is used to estimate the Weibull parameters and how (or if you even can) calculate the error for the Weibull estimation? A link/book or somewthing where I can look this up would be sufficient.

Answer 1

The "traditional two-parameter Weibull" model is presumably what you got from the lifelines package. That gives you a fully parameterized functional form for survival/reliability as a function of time.

For the error estimate used in the cited paper, Section 4.2 says:

The used fitting evaluation measure is the mean squared error (MSE) which will show the reliability fitting performance of the machine learning method. MSE computes the sum of the squares of the output differences at the $i^{th}$ time unit between the value of the approximate failure distribution and the output from the machine learning models...

The "approximate failure distribution" is presumably the Kaplan-Meier (KM) survival estimate, which gives a step-function estimate of survival as a function of time. The "output from the machine learning models" is the reliability/survival estimate from each model at each evaluated time point. So you just take the average of squared differences between observed (KM) and modeled survival values over a set of time points.

At least at first reading, however, it's not clear just which time points $i$ the authors included in their evaluations. That might be why you have trouble reproducing their results with the Weibull model. They could have used each individual time value along the time span of the data set, giving an overall, effectively integrated, estimate. Or they could have restricted evaluation to observed event times. The calculated MSE would differ depending on that choice of evaluation times.

Unlike for the machine-learning approaches, which use cross validation to optimize parameters for model fit, fitting a Weibull model does not require cross validation (CV) as that's done by maximum likelihood. You could do CV or bootstrapping, however, to evaluate the quality of the modeling process. That can give a sense of how well your model is likely to work on a new data sample.

If you choose to use the MSE survival/reliability criterion for that type of Weibull modeling, you would then fit a Weibull model on each "training" set (held-in cases in CV, bootstrap sample in bootstrap) and calculate the MSE against the corresponding KM curve of the "test" set (held-out cases in CV, entire data set in bootstrap). Multiple CV runs or bootstraps will give the most robust results.