I'm reading through the following paper (1). On page 2, section 2.1, there's a description of the selection of a beta distribution used:
Since both the choice of s and the strategy for handling the case that all values are above the threshold affect the result, we propose to abandon the use of a single absolute threshold and instead use a prior parameter distribution S given by P(si), where si , i = 1, . . . , N are possible thresholds. We use thresholds ranging from 0.01 to unity in steps of 0.01, i.e. N = 100. The distributions used in our experiments are Beta distributions with means 0.1, 0.15, 0.2 (α = 1 and β = 18, 11 + 1/3, 8) as shown in Fig 2
I think I understand the purpose - each of those 0.1, 0.15, 0.2 are the "important" value of the threshold, so I want the beta distribution to be highest at those increments, but I also want to test all values in the range 0.01-1.0
with a peak/mean at 0.1 or 0.15 or 0.2.
In the code (3), there are some beta distributions as hardcoded arrays of 100 values:
static float betaDist1[100] = {0.028911,0.048656,0.061306,0.068539,0.071703,0.071877,0.069915,0.066489,0.062117,0.057199,0.052034,0.046844,0.041786,0.036971,0.032470,0.028323,0.024549,0.021153,0.018124,0.015446,0.013096,0.011048,0.009275,0.007750,0.006445,0.005336,0.004397,0.003606,0.002945,0.002394,0.001937,0.001560,0.001250,0.000998,0.000792,0.000626,0.000492,0.000385,0.000300,0.000232,0.000179,0.000137,0.000104,0.000079,0.000060,0.000045,0.000033,0.000024,0.000018,0.000013,0.000009,0.000007,0.000005,0.000003,0.000002,0.000002,0.000001,0.000001,0.000001,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000};
static float betaDist2[100] = {0.012614,0.022715,0.030646,0.036712,0.041184,0.044301,0.046277,0.047298,0.047528,0.047110,0.046171,0.044817,0.043144,0.041231,0.039147,0.036950,0.034690,0.032406,0.030133,0.027898,0.025722,0.023624,0.021614,0.019704,0.017900,0.016205,0.014621,0.013148,0.011785,0.010530,0.009377,0.008324,0.007366,0.006497,0.005712,0.005005,0.004372,0.003806,0.003302,0.002855,0.002460,0.002112,0.001806,0.001539,0.001307,0.001105,0.000931,0.000781,0.000652,0.000542,0.000449,0.000370,0.000303,0.000247,0.000201,0.000162,0.000130,0.000104,0.000082,0.000065,0.000051,0.000039,0.000030,0.000023,0.000018,0.000013,0.000010,0.000007,0.000005,0.000004,0.000003,0.000002,0.000001,0.000001,0.000001,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000};
static float betaDist3[100] = {0.006715,0.012509,0.017463,0.021655,0.025155,0.028031,0.030344,0.032151,0.033506,0.034458,0.035052,0.035331,0.035332,0.035092,0.034643,0.034015,0.033234,0.032327,0.031314,0.030217,0.029054,0.027841,0.026592,0.025322,0.024042,0.022761,0.021489,0.020234,0.019002,0.017799,0.016630,0.015499,0.014409,0.013362,0.012361,0.011407,0.010500,0.009641,0.008830,0.008067,0.007351,0.006681,0.006056,0.005475,0.004936,0.004437,0.003978,0.003555,0.003168,0.002814,0.002492,0.002199,0.001934,0.001695,0.001481,0.001288,0.001116,0.000963,0.000828,0.000708,0.000603,0.000511,0.000431,0.000361,0.000301,0.000250,0.000206,0.000168,0.000137,0.000110,0.000088,0.000070,0.000055,0.000043,0.000033,0.000025,0.000019,0.000014,0.000010,0.000007,0.000005,0.000004,0.000002,0.000002,0.000001,0.000001,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000};
static float betaDist4[100] = {0.003996,0.007596,0.010824,0.013703,0.016255,0.018501,0.020460,0.022153,0.023597,0.024809,0.025807,0.026607,0.027223,0.027671,0.027963,0.028114,0.028135,0.028038,0.027834,0.027535,0.027149,0.026687,0.026157,0.025567,0.024926,0.024240,0.023517,0.022763,0.021983,0.021184,0.020371,0.019548,0.018719,0.017890,0.017062,0.016241,0.015428,0.014627,0.013839,0.013068,0.012315,0.011582,0.010870,0.010181,0.009515,0.008874,0.008258,0.007668,0.007103,0.006565,0.006053,0.005567,0.005107,0.004673,0.004264,0.003880,0.003521,0.003185,0.002872,0.002581,0.002312,0.002064,0.001835,0.001626,0.001434,0.001260,0.001102,0.000959,0.000830,0.000715,0.000612,0.000521,0.000440,0.000369,0.000308,0.000254,0.000208,0.000169,0.000136,0.000108,0.000084,0.000065,0.000050,0.000037,0.000027,0.000019,0.000014,0.000009,0.000006,0.000004,0.000002,0.000001,0.000001,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000};
Now I'm trying to actually generate those arrays, by plugging in the values (a = 1.0, b = 8.0/11.3333/18.0)
into the Python scipy.stats.beta
(4) module:
#!/usr/bin/env python3
from scipy.stats import beta
import sys
a = 1.0
bs = [8.0, 11.33333333, 18.0]
beta_distrib_1 = [[] for _ in range(3)]
beta_distrib_2 = [[] for _ in range(3)]
for i in range(100):
for j, b in enumerate(bs):
beta_distrib_1[j].append(beta.pdf(i, a, b))
beta_distrib_2[j].append(beta.pdf(i, a, b))
print(beta_distrib_1)
print(beta_distrib_2)
However, the output is not even close to those hardcoded arrays. Could somebody describe how to go from a=1.0,b=8.0
to those actual arrays of values? Should I be using the pdf
function? What's the parameter x
? Say I want to do something similar, but with 20 values instead of 100.
Additionally, if I use beta(1.0, 8.0)
, beta(1.0, 11.333)
, beta(1.0, 18.0)
, the means are not 0.1, 0.15, and 0.20 in scipy.
Also, my ultimate goal is to use the Boost C++ beta distribution library (5) - but I assume that once I understand the math behind the idea, I'll be able to proceed with any library in any language
pdf(x, a, b)
with x in the range of N, to generate N values of a beta distribution correct? $\endgroup$betaDist1
etc. lies: exactly which values of $x$ do these arrays correspond to? Possibilities include $0,0.01, \ldots, 0.99;$ $0.01,0.02,\ldots,1.00;$ $0.005,0.015,.\ldots,0.995;$ and $0,1/99,2/99,\ldots,1.$ And are the values intended to be the pdf at each $x$ or would they be the integrated probabilities of each of the little intervals? These reveal some of the many problems of using undocumented scientific code. Best to develop your own from scratch. $\endgroup$