0
$\begingroup$

From thinkstats2 the O'Reilly text.

class cdf(object):

    def ValueArray(self, ps):
      """Returns InverseCDF(p), the value that corresponds to probability p.

      Args:
          ps: NumPy array of numbers in the range [0, 1]

      Returns:
          NumPy array of values
      """
      ps = np.asarray(ps)
      if np.any(ps < 0) or np.any(ps > 1):
          raise ValueError('Probability p must be in range [0, 1]')

      index = np.searchsorted(self.ps, ps, side='left')
      return self.xs[index]

     def Sample(self, n):
        """Generates a random sample from this distribution.

        n: int length of the sample
        returns: NumPy array
        """
        ps = np.random.random(n) # chooses n random terms in [0,1)
        return self.ValueArray(ps) 
        # so here i choose an 

        # ps are defined as the % of getting <= a given term in the cdf
        # ValueArray returns the val associated with a % <= desired

def ResampleRowsWeighted(df, column='finalwgt'): 
   weights = df[column] # giving us the desired weights
   cdf = Cdf(dict(weights)) # creating a cdf object
   indices = cdf.Sample(len(weights)) # returns list of vals
   sample = df.loc[indices] # returns chosen rows
   return sample 

why do we use a cdf instead of pdf?

with a cdf were most likely to land in the biggest gap, the largest difference between 2 percentiles

with a pmf we can use a series and add each perc together, and then pick a random number in [0,1) where larger weights are more likely to get picked

$\endgroup$
3
  • 2
    $\begingroup$ Can you explain in words what this code is doing? Not everyone reading this will be a Python programmer. $\endgroup$ – dsaxton Jul 21 '15 at 0:33
  • $\begingroup$ ResampleRowsWeighted, first assigns the weights from a specified columns that does so. Then creates a cdf of the weights wrt to the index of the data frame. Where each weight is mapped to the percentile associated with that weight. Then we assign our indices to be the index associated with the closest percentile to the right of each random percentile chosen in Sample. Then we pull a sample of rows from our df consisting of the chosen index and return this. $\endgroup$ – Brydon Parker Jul 21 '15 at 0:53
  • 1
    $\begingroup$ Among others, please see stats.stackexchange.com/questions/77845/… and stats.stackexchange.com/questions/65096/…. $\endgroup$ – whuber Jul 21 '15 at 1:20
1
$\begingroup$

Not sure I completely follow the description, but presumably the program is just using the probability transform.

The idea here is that if $F$ is a distribution function and we define $F^{-1}(u) \equiv \inf \{ x : F(x) \geq u \}$, then $F^{-1}(U)$ is a realization of a random variable with distribution $F$ when $U \sim$ uniform$(0, 1)$. The reason the distribution function is used is because of this result.

If we wanted to use the mass function instead, we would have to "chop up" the unit interval $(0, 1)$ into slices with lengths equal to the probabilities associated with that mass function and then map these to the values of the random variable. This is a lot of work and it turns out to just be a manual way of calculating $F^{-1}(U)$.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.