2
$\begingroup$

I've done some search for similar questions, but they're not the same as what I'm trying to get.

Assume that there's a server that handles requests $r$ and returns a set of items $I_{r}$ of random length $l_{r}$. There won't be two identical items in one set. The length of the returned sets could be conformed to some distribution but I'm not going to specify it here.

Currently, I know about $N$ items that could be returned. And I will make $M$ requests, which means I will get $M$ sets of items back. Some returned items may be the ones I have known, and others are not. After finishing the requests, I would have seen some items that I didn't know about. Also, not all the items that I knew would appear in the $M$ returned sets of items.

The issue that would make the calculation difficult is that the items are not returned with equal probability. I do not know the probability of each item, though to make it easier, it could be assumed that the frequency of an item in the $M$ requests is proportional to the probability that it would be returned. (It would be great if somehow this assumption would be held under some confidence level).

Similarly, it could be assumed the length distribution of the returned set would be the same as the $M$ returned sets. In fact, I'm not sure if this matters. It could also be equivalent to that the server returns one item each time and I make more requests, can it?

How can I infer the total number of items on the server based on the information above?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

You are looking for this: https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_variance where you should use reliability weights. You will have to make some estimate of the item probabilities, though. Imagine what happens if the first half of the items have zero probability and the others have probability 1 - the variance will be radically different from the case where it's the other way around.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Not sure if I understood correctly. So I'll consider the ratio of known addons in each request answer as $x_{i}$. And compute its mean and variance with reliability weights, where the assumption holds as the frequency of an item in the 𝑀 requests is proportional to the probability that it would be returned. After that, get the range of the ratio $x$ of known addons under some confidence level. Then use $N / x$ to infer the total number of items? $\endgroup$
    – Awdrtg
    Commented Apr 2, 2020 at 4:45
  • $\begingroup$ the title of your question asks how to calculate confidence intervals with weighted data, so i answered that question. do you have a different question? if so, consider editing your question or creating a new one, and also consider reviewing stackoverflow.com/help/how-to-ask $\endgroup$
    – Dave Kielpinski
    Commented Apr 2, 2020 at 5:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.