1
$\begingroup$

I'm reading through the following paper (1). On page 2, section 2.1, there's a description of the selection of a beta distribution used:

Since both the choice of s and the strategy for handling the case that all values are above the threshold affect the result, we propose to abandon the use of a single absolute threshold and instead use a prior parameter distribution S given by P(si), where si , i = 1, . . . , N are possible thresholds. We use thresholds ranging from 0.01 to unity in steps of 0.01, i.e. N = 100. The distributions used in our experiments are Beta distributions with means 0.1, 0.15, 0.2 (α = 1 and β = 18, 11 + 1/3, 8) as shown in Fig 2

Here's fig 2: beta distribution

I think I understand the purpose - each of those 0.1, 0.15, 0.2 are the "important" value of the threshold, so I want the beta distribution to be highest at those increments, but I also want to test all values in the range 0.01-1.0 with a peak/mean at 0.1 or 0.15 or 0.2.

In the code (3), there are some beta distributions as hardcoded arrays of 100 values:

static float betaDist1[100] = {0.028911,0.048656,0.061306,0.068539,0.071703,0.071877,0.069915,0.066489,0.062117,0.057199,0.052034,0.046844,0.041786,0.036971,0.032470,0.028323,0.024549,0.021153,0.018124,0.015446,0.013096,0.011048,0.009275,0.007750,0.006445,0.005336,0.004397,0.003606,0.002945,0.002394,0.001937,0.001560,0.001250,0.000998,0.000792,0.000626,0.000492,0.000385,0.000300,0.000232,0.000179,0.000137,0.000104,0.000079,0.000060,0.000045,0.000033,0.000024,0.000018,0.000013,0.000009,0.000007,0.000005,0.000003,0.000002,0.000002,0.000001,0.000001,0.000001,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000};

static float betaDist2[100] = {0.012614,0.022715,0.030646,0.036712,0.041184,0.044301,0.046277,0.047298,0.047528,0.047110,0.046171,0.044817,0.043144,0.041231,0.039147,0.036950,0.034690,0.032406,0.030133,0.027898,0.025722,0.023624,0.021614,0.019704,0.017900,0.016205,0.014621,0.013148,0.011785,0.010530,0.009377,0.008324,0.007366,0.006497,0.005712,0.005005,0.004372,0.003806,0.003302,0.002855,0.002460,0.002112,0.001806,0.001539,0.001307,0.001105,0.000931,0.000781,0.000652,0.000542,0.000449,0.000370,0.000303,0.000247,0.000201,0.000162,0.000130,0.000104,0.000082,0.000065,0.000051,0.000039,0.000030,0.000023,0.000018,0.000013,0.000010,0.000007,0.000005,0.000004,0.000003,0.000002,0.000001,0.000001,0.000001,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000};

static float betaDist3[100] = {0.006715,0.012509,0.017463,0.021655,0.025155,0.028031,0.030344,0.032151,0.033506,0.034458,0.035052,0.035331,0.035332,0.035092,0.034643,0.034015,0.033234,0.032327,0.031314,0.030217,0.029054,0.027841,0.026592,0.025322,0.024042,0.022761,0.021489,0.020234,0.019002,0.017799,0.016630,0.015499,0.014409,0.013362,0.012361,0.011407,0.010500,0.009641,0.008830,0.008067,0.007351,0.006681,0.006056,0.005475,0.004936,0.004437,0.003978,0.003555,0.003168,0.002814,0.002492,0.002199,0.001934,0.001695,0.001481,0.001288,0.001116,0.000963,0.000828,0.000708,0.000603,0.000511,0.000431,0.000361,0.000301,0.000250,0.000206,0.000168,0.000137,0.000110,0.000088,0.000070,0.000055,0.000043,0.000033,0.000025,0.000019,0.000014,0.000010,0.000007,0.000005,0.000004,0.000002,0.000002,0.000001,0.000001,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000};

static float betaDist4[100] = {0.003996,0.007596,0.010824,0.013703,0.016255,0.018501,0.020460,0.022153,0.023597,0.024809,0.025807,0.026607,0.027223,0.027671,0.027963,0.028114,0.028135,0.028038,0.027834,0.027535,0.027149,0.026687,0.026157,0.025567,0.024926,0.024240,0.023517,0.022763,0.021983,0.021184,0.020371,0.019548,0.018719,0.017890,0.017062,0.016241,0.015428,0.014627,0.013839,0.013068,0.012315,0.011582,0.010870,0.010181,0.009515,0.008874,0.008258,0.007668,0.007103,0.006565,0.006053,0.005567,0.005107,0.004673,0.004264,0.003880,0.003521,0.003185,0.002872,0.002581,0.002312,0.002064,0.001835,0.001626,0.001434,0.001260,0.001102,0.000959,0.000830,0.000715,0.000612,0.000521,0.000440,0.000369,0.000308,0.000254,0.000208,0.000169,0.000136,0.000108,0.000084,0.000065,0.000050,0.000037,0.000027,0.000019,0.000014,0.000009,0.000006,0.000004,0.000002,0.000001,0.000001,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000};

Now I'm trying to actually generate those arrays, by plugging in the values (a = 1.0, b = 8.0/11.3333/18.0) into the Python scipy.stats.beta (4) module:

#!/usr/bin/env python3

from scipy.stats import beta
import sys

a = 1.0
bs = [8.0, 11.33333333, 18.0]

beta_distrib_1 = [[] for _ in range(3)]
beta_distrib_2 = [[] for _ in range(3)]

for i in range(100):
    for j, b in enumerate(bs):
        beta_distrib_1[j].append(beta.pdf(i, a, b))
        beta_distrib_2[j].append(beta.pdf(i, a, b))

print(beta_distrib_1)
print(beta_distrib_2)

However, the output is not even close to those hardcoded arrays. Could somebody describe how to go from a=1.0,b=8.0 to those actual arrays of values? Should I be using the pdf function? What's the parameter x? Say I want to do something similar, but with 20 values instead of 100.

Additionally, if I use beta(1.0, 8.0), beta(1.0, 11.333), beta(1.0, 18.0), the means are not 0.1, 0.15, and 0.20 in scipy.

Also, my ultimate goal is to use the Boost C++ beta distribution library (5) - but I assume that once I understand the math behind the idea, I'll be able to proceed with any library in any language

$\endgroup$
4
  • $\begingroup$ It can't be done because those hard-coded arrays aren't what they claim to be. Eg, the first is a close approximation to the pdf of a Beta$(1.83, 17.36)$ distribution at points spaced by $0.01$ (although it's hard to determine what their origin is, because there are only 100 of them instead of 101). Since the pdf of a Beta distribution is simple to compute (and you can obtain the probabilities over small intervals using a trapozoidal or Simpson's Rule calculation), why not just code it correctly yourself? BTW, stats.stackexchange.com/questions/12232 shows how to find the parameters. $\endgroup$
    – whuber
    Mar 20, 2019 at 20:47
  • $\begingroup$ Thanks - I'll use those formulae. Is my logic of applying pdf(x, a, b) with x in the range of N, to generate N values of a beta distribution correct? $\endgroup$
    – Sevag
    Mar 21, 2019 at 13:52
  • $\begingroup$ Your code misses the mark because the Beta pdf is defined only for arguments $x$ between $0$ and $1.$ That's where the ambiguity in betaDist1 etc. lies: exactly which values of $x$ do these arrays correspond to? Possibilities include $0,0.01, \ldots, 0.99;$ $0.01,0.02,\ldots,1.00;$ $0.005,0.015,.\ldots,0.995;$ and $0,1/99,2/99,\ldots,1.$ And are the values intended to be the pdf at each $x$ or would they be the integrated probabilities of each of the little intervals? These reveal some of the many problems of using undocumented scientific code. Best to develop your own from scratch. $\endgroup$
    – whuber
    Mar 21, 2019 at 14:03
  • $\begingroup$ Thanks again for your help. If you reply as an answer, I'll select it as best answer. $\endgroup$
    – Sevag
    Mar 22, 2019 at 12:59

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.