2
$\begingroup$

I would like to understand more about how to use the uncertainty in a sample proportion.

Imagine I need to offer a warranty for a product against individual sold items failing to work as advertised. I would like to estimate the cost of offering the warranty based on the track record of the same products already sold. As I go along, selling a few more products and learning about more failures and paying out on the warranty, I will adjust my model of what the warranty is costing me.

Rather than delve into the depths of the various methods that I could use I want to understand more on how to frame the analytical objective.


Lets imagine the proportion of failed products todate is 5% and that I have some measure of uncertainty for the proportion, which happens to be 4%. I need a single number answer, i.e. how much I should roll into the cost of the product as a per item warranty cost.

I think I can choose to either:

  • take into account the uncertainty as part of the process of establishing the warranty cost, e.g. 5% +- 4% = a range from 1% to 9%,

OR

  • assume that for this type of problem the central proportion, 5%, is sufficient description of the liability, given that I will be continutally resetting my calculation as I go along

In the first case as the range is so wide it seems rather arbitrary to add the proportion and uncertainty and thus establish the cost of the warranty as 9% and in this case we can see that it nearly doubles the cost compared to the basic proportion.

I'd have a similar worry with the basic proportion of the second case or picking any other single number. Yet a single number is what I need and I have a nagging feeling that I am ignoring something important, hence I don't really know how to use the information.

It might be that my choice of data collection period for establishng the running proportion and the accounting period for the latest estimate of the cost of the warranty should be linked, but I don't want to pressuppose that.


EDIT

Additional context:

  • There is no fixed underlying failure rate, it can evolve with time.

  • There is nothing that says that there should be any relation between failure modes and their relative frequencies, solutions and further problems introduced by fixing the previous faults.

  • There is nothing that says that next year should bear any resemblence to last year, nor change smoothly etc

.. and yet there is a need to prevent the organisation selling the products from going bust through not making provision for warranties or, equally, going bust from charging too much for its products.

I haven't defined many of the terms that statisticians might like to, its deliberate because its all to easy to disappear down a rabbit hole of definitions. If it helps, I've eased away from central "proportion" to "estimate" so as to disengage from the frequentist view. I think the concern about accounting for uncertainty is that this itself could bias (non statisticians meaning of the word "bias") the result and from the simple, non-statistician's view, the uncertainty could go either way and be just as damaging to the business either way.

I've realised I am playing the devil's advocate with my own beliefs but am looking for some further convincing.

Refined question: Are there any features of this scenario that mean that the central estimate is not the best one?

$\endgroup$

1 Answer 1

1
$\begingroup$

Rather than delve into the depths of the various methods that I could use I want to understand more on how to frame the analytical objective.

Actually, it turns out that one of the key parts for understanding the various methods (both for this specific problem and for statistics in general) is understanding the objective.

You say that "I have some measure of uncertainty for the proportion, which happens to be 4%". But what exactly do you mean? Four percent of what? Looking at your first bullet point, you seem to mean that 4% is half the width of a certain statistical interval for the estimate. But there are several kinds of statistical intervals, such as confidence intervals, prediction intervals, tolerance intervals, and credible intervals, which differ not only in how they're computed but what information they're supposed to give you. Furthermore, statistical intervals tend to also created according to a numeric specification of their own; for example, you could choose to construct an 80% confidence interval or a 95% confidence interval.

Your second bullet point proposes to, basically, ignore all uncertainty in the estimate. I don't think that's a good way to go. Your intuition that uncertainty should be tracked is a good one. You could still account for uncertainty in a single estimate, though, with Bayesian methods: you would quote not the observed frequency of failure but the posterior probability of failure, which adjusts the observed frequency according to the sample size and your prior beliefs.

I think your next step should be to read a little about confidence intervals and Bayesian inference to get a better sense of how uncertainty can be quantified.

$\endgroup$
3
  • $\begingroup$ thank you for your starting thoughts. I've realised as a result I haven't put the right emphasis in the question. I will give it a day's thought and see if I can clarify. $\endgroup$
    – Puffin
    Oct 20, 2015 at 23:25
  • $\begingroup$ Reply to your edit: which estimate is best depends on your notion of "best". See, for example: en.wikipedia.org/wiki/Loss_function $\endgroup$ Oct 24, 2015 at 12:40
  • $\begingroup$ Thank you again, that last link you posted has helped a great deal. Obviously there's a lot there but just realising that there are formal methods for including the range of loss outcomes is a big step forwards. $\endgroup$
    – Puffin
    Oct 26, 2015 at 19:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.