2
$\begingroup$

I have the views on the top 100 videos using a tag in TikTok and want to estimate the total number of videos in that tag. I know the distribution for other tags so I can make a guess as to what it is for this one, but I don't know for sure.

One formal model of this: I'm given the $k$ largest of $n$ i.i.d. samples, and I wish to estimate $n$.

Let $F(v)=p(X\leq v)$. Then I believe that the probability of having exactly $k$ of $n$ samples greater than $v$ is binomially distributed as $B(n, k, F(v))$.

So I think one way I could solve this is to set $v$ to be the smallest of the $k$ samples, and then find the value of $n$ which maximizes the likelihood. Is this the best way to solve the problem? I feel like I'm not using all of the relevant information.

Also, empirically it doesn't seem to give a very good estimate: you can see that there is some correlation here, but it's pretty weak:

enter image description here

Code to produce image:

import matplotlib.pyplot as plt
import numpy as np
from scipy.stats import norm
from scipy import stats
from scipy import optimize
from scipy.special import comb
import pandas as pd

def run_test(size, k = 10):
    data = norm.rvs(size = size)
    biggest = sorted(data)[-k]

    def score(n):
        l = 1 - norm.cdf(biggest)
        return -binom.logpmf(k, n, l)
    
    fitted = optimize.minimize(score, x0 = 10, method = 'Nelder-Mead',
               options={'xatol': 1e-8})
    return fitted.x

sizes = np.random.randint(low = 100, high = 10000, size = 500)
estimates = [run_test(s) for s in sizes]
plt.scatter(sizes, estimates)
plt.xlabel('true size')
plt.ylabel('estimated size')
$\endgroup$
  • $\begingroup$ What is the practical context? Do you know the true distribution function $F$? $\endgroup$ – kjetil b halvorsen Oct 26 at 17:11
  • $\begingroup$ I have the views on the top 100 videos using a tag in TikTok and want to estimate the total number of videos in that tag. I know the distribution for other tags so I can make a guess as to what it is for this one, but I don't know for sure. $\endgroup$ – Xodarap Oct 26 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.