I am looking for an appropriate method to analyse these data/situation: We sell electronic devices. Customers are encouraged to report defective devices. Around every months we calculate the cumulative rate of {defective_devices}/{all_sold_devices}. I would like to monitor for sharp increases in the rates of defectives; for example if I calculate the cumulative rates each months from 1 to 5 I will have 0.3, 0.55, 0.4, 0.78, 0.85... it appears that the slope of the curve sharply increases at 0.78 onwards. What statistical methods can I use to determine that the slope significantly increases by a certain predetermined value? I'm also worried about the arbitrary selection of monitoring every month, why not every week or even every day? How does one determine what is the proper monitoring period?
2 Answers
Rather than rates , you should be modelling the number of complaints as a function of the number sold perhaps including previous sales as complaints might have some time dependency. One needs to possibly identify level shifts and/or trends in the number of complaints. Seasonal structure might also have to be identified as that could be a source of variation. If you are using daily data, there could be a holiday effect. Care should be taken not to be blind-sided by unusual values when constructing your model. In this way you will be able to predict and then to detect when things have chnaged. The tools that you are after might be available to you while searching for "automatic transfer function intervention detection" . Rates are used when one doesn't know how to model the observed two variables. Forecasts can always be turned into rates if that is useful to you but one doesn't start with rates as you have effectively collapsed two columns of data into one, with resultant loss of information.
-
$\begingroup$ I think we are on the same wavelength. I also think that interval rates throw out too much information. But I approach the time dependence through survival analysis. Right censoring could be an issue that you don't uncover through time series analysis. The big issue from my experience is underreporting which I think can only be remedied by sampling (tracking very closely a random sample of the items). How large a sample to use is a cost versus estimation quality tradeoff. $\endgroup$ Commented Oct 2, 2012 at 20:19
-
$\begingroup$ @MichaelChernick underreporting is generally a problem but if you assume that it is rather constant over time then looking for shifts in trends/rates of reporting would not be affected by underreporting. $\endgroup$ Commented Oct 2, 2012 at 20:27
-
$\begingroup$ @francogrex We have a different perspective I guess. We want to know the reliability of our product as best as we can. This is both for our product performance reports that are used by the sales people to get the physicians to use our products over our big competitor. I worked at St Jude Medical whose rival in Minneapolis was/is Medtronic. Also postmarket surveillance is very important to the FDA. Basically absolute performance was the key for us and not just some measure of relative performance. $\endgroup$ Commented Oct 2, 2012 at 20:36
We had a similar problem when I worked in the medical device industry. We wanted to determine the reliability of our product in my case a pacemaker. Physicians were encouraged to report failures that are detected when they explant the device. Most failure are battery depletions but other modes of failure occur. We used the Kaplan-Meier estimate of time to failure as our performance measure perhaps focussing on its value at 5 years post implant. Reports are coming in continuously. So we don't have the problem of picking arbitrary intervals. This is a different approach to the same problem but it does have valid statistical methods for characterizing the failure time distribution. A parametric approach which would involve say fitting a Weibull distribution to the reporting data would provide you with a function that describes how the failure rate increases or decreases with time given the estimated parameters.
But the real underlying problem we had and you might as well is underreporting. Although the physicians are supposed to report the failures back to the manufacturer or better yet return the explanted device to the manufacturers so that the manufacturer's engineers can diagnose the failure they don't always to it. Underreporting rates were thought to be as high as 30%. You can have a very large bias (estimating that your failure rate is lower than it is in reality at any given time since implant).
The FDA had a plan for postmarket surveillance that would adjust for this bias based on independent detailed tracking of a random sample of the implants. I published a paper in the DIA Journal that showed that the adjustment depended heavily on some assumptions and overcorrection was possible. However in principle a statistical methodology for this would be to fit a parametric survival curve and adjust the parameter estimates based on random sampling the devices.
