Skip to main content
Title
Link
MånsT
  • 12.1k
  • 1
  • 50
  • 66

How to estimate storage needs using the PERT Distributiondistribution for filesizes? ¿How How to aggregate them without falling ininto the Flawflaw of Extremesextremes?

added 674 characters in body; edited tags
Source Link
Luxspes
  • 301
  • 2
  • 10

How to estimate storage needs? I am using the PERT Distribution for filesizes ¿How? ¿How to aggregate itthem without falling in the Flaw of Extremes?

Lets say I know that I am going to store the information of 10,000 people each year for 4 years, that is 40,000 files. Now If I estimate that on the best case scenario the information from each person is going to weight .25MBytes, I believe there is a 50% chance that it will weight 1Mbyte, and I believe that the worst case scenario is that each person will require 5Mbytes...

Would it be ok for me to use the PERT formula (O + 4M + P)/6 and conclude ((.25 + 4(1) +5)/6 = 1.54) that my storage needs will be 40,000x1.5 = 60,000 Mbytes = 60 Gigabytes ?

I am trying to reach the answers "How likely would that be? ¿How can I know if that is X% likely?" numbers using Mathematica, so far it seems I can answer: How likely would that be? by writing this:

$Probability[x < 1.54, x \[Distributed] PERTDistribution[{0.25, 5}, 1]]$

I get a 0.55 probability of X being smaller than 1.54. ¿Correct?

I am asking this because I need to estimate the hardware requirements for a software system, and all the information I am given is that "I am going to store the information of 10,000 people each year for 4 years, that is 40,000 files". Once the system works for some months I will be able to use its behavior to predict its needs in a more accurately way, but in the meanwhile, I need to be able to give an initial estimate... and I am not sure on how to do that...

So. would it be safe to say that I have a 0.55 probability of my storage needs being smaller than 60 Gigabytes?

I am having some trouble beliving this is right, so I made the follwing experiment:

$Table[Fold[Plus, 0, RandomVariate[PERTDistribution[{.25, 5}, 1], 40000]], {10000}] $

To simulate the total size that I would need and, the thing is that this giving me a different answer than I would think:

If I do this:

$Quantile[Table[Fold[Plus, 0, RandomVariate[PERTDistribution[{.25, 5}, 1], 40000]], {10000}], 95/100]$

Answer: $61930.7$

It should give me the same answer as this (Should'nt it?) why it doesn't?

$(Quantile[PERTDistribution[{.25, 5}, 1], 95/100]) (40000)$

Answer: $121,938$

But it does not, it seems like estimating a single files size with 95% confidence and then multiplying it for 40,000 is very different... ¿where is

Looking at my statistics books, this looks like the mistake in chapter 17 in "The Flaw of Averages" by by Dr. Sam Savage "The Flaw of Extremes" "In bottom - up budgeting, reporting the 90th percentile of cash needs leads to ever thicker layers of unnecessary cash as the figures are rolled up to higher levels. Even more harmful things result from focusing on above - or below - average results, such as test scores or health - related statistics."

I think that is why there is a difference, in this case, it is not bottom - up budgeting, but it is bottom - up file storage estimation, and the same rules should apply ¿no? also ¿is

¿Is there a way to deal with this mistake "symbolically" that is, without having to wait for slow simulated generation of sizes?

How to estimate storage needs? I am using the PERT Distribution for filesizes ¿How to aggregate it?

Lets say I know that I am going to store the information of 10,000 people each year for 4 years, that is 40,000 files. Now If I estimate that on the best case scenario the information from each person is going to weight .25MBytes, I believe there is a 50% chance that it will weight 1Mbyte, and I believe that the worst case scenario is that each person will require 5Mbytes...

Would it be ok for me to use the PERT formula (O + 4M + P)/6 and conclude ((.25 + 4(1) +5)/6 = 1.54) that my storage needs will be 40,000x1.5 = 60,000 Mbytes = 60 Gigabytes ?

I am trying to reach the answers "How likely would that be? ¿How can I know if that is X% likely?" numbers using Mathematica, so far it seems I can answer: How likely would that be? by writing this:

$Probability[x < 1.54, x \[Distributed] PERTDistribution[{0.25, 5}, 1]]$

I get a 0.55 probability of X being smaller than 1.54. ¿Correct?

I am asking this because I need to estimate the hardware requirements for a software system, and all the information I am given is that "I am going to store the information of 10,000 people each year for 4 years, that is 40,000 files". Once the system works for some months I will be able to use its behavior to predict its needs in a more accurately way, but in the meanwhile, I need to be able to give an initial estimate... and I am not sure on how to do that...

So. would it be safe to say that I have a 0.55 probability of my storage needs being smaller than 60 Gigabytes?

I am having some trouble beliving this is right, so I made the follwing experiment:

$Table[Fold[Plus, 0, RandomVariate[PERTDistribution[{.25, 5}, 1], 40000]], {10000}] $

To simulate the total size that I would need and, the thing is that this giving me a different answer than I would think:

If I do this:

$Quantile[Table[Fold[Plus, 0, RandomVariate[PERTDistribution[{.25, 5}, 1], 40000]], {10000}], 95/100]$

Answer: $61930.7$

It should give me the same answer as this (Should'nt it?) why it doesn't?

$(Quantile[PERTDistribution[{.25, 5}, 1], 95/100]) (40000)$

Answer: $121,938$

But it does not, it seems like estimating a single files size with 95% confidence and then multiplying it for 40,000 is very different... ¿where is the mistake? also ¿is there a way to deal with this mistake "symbolically" that is, without having to wait for slow generation of sizes?

How to estimate storage needs using the PERT Distribution for filesizes? ¿How to aggregate them without falling in the Flaw of Extremes?

Lets say I know that I am going to store the information of 10,000 people each year for 4 years, that is 40,000 files. Now If I estimate that on the best case scenario the information from each person is going to weight .25MBytes, I believe there is a 50% chance that it will weight 1Mbyte, and I believe that the worst case scenario is that each person will require 5Mbytes...

Would it be ok for me to use the PERT formula (O + 4M + P)/6 and conclude ((.25 + 4(1) +5)/6 = 1.54) that my storage needs will be 40,000x1.5 = 60,000 Mbytes = 60 Gigabytes ?

I am trying to reach the answers "How likely would that be? ¿How can I know if that is X% likely?" numbers using Mathematica, so far it seems I can answer: How likely would that be? by writing this:

$Probability[x < 1.54, x \[Distributed] PERTDistribution[{0.25, 5}, 1]]$

I get a 0.55 probability of X being smaller than 1.54. ¿Correct?

I am asking this because I need to estimate the hardware requirements for a software system, and all the information I am given is that "I am going to store the information of 10,000 people each year for 4 years, that is 40,000 files". Once the system works for some months I will be able to use its behavior to predict its needs in a more accurately way, but in the meanwhile, I need to be able to give an initial estimate... and I am not sure on how to do that...

So. would it be safe to say that I have a 0.55 probability of my storage needs being smaller than 60 Gigabytes?

I am having some trouble beliving this is right, so I made the follwing experiment:

$Table[Fold[Plus, 0, RandomVariate[PERTDistribution[{.25, 5}, 1], 40000]], {10000}] $

To simulate the total size that I would need and, the thing is that this giving me a different answer than I would think:

If I do this:

$Quantile[Table[Fold[Plus, 0, RandomVariate[PERTDistribution[{.25, 5}, 1], 40000]], {10000}], 95/100]$

Answer: $61930.7$

It should give me the same answer as this (Should'nt it?) why it doesn't?

$(Quantile[PERTDistribution[{.25, 5}, 1], 95/100]) (40000)$

Answer: $121,938$

But it does not, it seems like estimating a single files size with 95% confidence and then multiplying it for 40,000 is very different...

Looking at my statistics books, this looks like the mistake in chapter 17 in "The Flaw of Averages" by by Dr. Sam Savage "The Flaw of Extremes" "In bottom - up budgeting, reporting the 90th percentile of cash needs leads to ever thicker layers of unnecessary cash as the figures are rolled up to higher levels. Even more harmful things result from focusing on above - or below - average results, such as test scores or health - related statistics."

I think that is why there is a difference, in this case, it is not bottom - up budgeting, but it is bottom - up file storage estimation, and the same rules should apply ¿no?

¿Is there a way to deal with this "symbolically" that is, without having to wait for slow simulated generation of sizes?

added 775 characters in body
Source Link
Luxspes
  • 301
  • 2
  • 10

How to estimate storage needs? IsI am using the PERT Distribution a valid wayfor filesizes ¿How to doaggregate it?

Lets say I know that I am going to store the information of 10,000 people each year for 4 years, that is 40,000 files. Now If I estimate that on the best case scenario the information from each person is going to weight .25MBytes, I believe there is a 50% chance that it will weight 1Mbyte, and I believe that the worst case scenario is that each person will require 5Mbytes...

Would it be ok for me to use the PERT formula (O + 4M + P)/6 and conclude ((.25 + 4(1) +5)/6 = 1.54) that my storage needs will be 40,000x1.5 = 60,000 Mbytes = 60 Gigabytes ?

I am trying to reach the answers "How likely would that be? ¿How can I know if that is X% likely?" numbers using Mathematica, so far it seems I can answer: How likely would that be? by writing this:

Probability[x < 1.54, x [Distributed] PERTDistribution[{0.25, 5}, 1]]$Probability[x < 1.54, x \[Distributed] PERTDistribution[{0.25, 5}, 1]]$

I get a 0.55 probability of X being smaller than 1.54. ¿Correct?

I am asking this because I need to estimate the hardware requirements for a software system, and all the information I am given is that "I am going to store the information of 10,000 people each year for 4 years, that is 40,000 files". Once the system works for some months I will be able to use its behavior to predict its needs in a more accurately way, but in the meanwhile, I need to be able to give an initial estimate... and I am not sure on how to do that...

So. would it be safe to say that I have a 0.55 probability of my storage needs being smaller than 60 Gigabytes?

I am having some trouble beliving this is right, so I made the follwing experiment:

$Table[Fold[Plus, 0, RandomVariate[PERTDistribution[{.25, 5}, 1], 40000]], {10000}] $

To simulate the total size that I would need and, the thing is that this giving me a different answer than I would think:

If I do this:

$Quantile[Table[Fold[Plus, 0, RandomVariate[PERTDistribution[{.25, 5}, 1], 40000]], {10000}], 95/100]$

Answer: $61930.7$

It should give me the same answer as this (Should'nt it?) why it doesn't?

$(Quantile[PERTDistribution[{.25, 5}, 1], 95/100]) (40000)$

Answer: $121,938$

But it does not, it seems like estimating a single files size with 95% confidence and then multiplying it for 40,000 is very different... ¿where is the mistake? also ¿is there a way to deal with this mistake "symbolically" that is, without having to wait for slow generation of sizes?

How to estimate storage needs? Is PERT Distribution a valid way to do it?

Lets say I know that I am going to store the information of 10,000 people each year for 4 years, that is 40,000 files. Now If I estimate that on the best case scenario the information from each person is going to weight .25MBytes, I believe there is a 50% chance that it will weight 1Mbyte, and I believe that the worst case scenario is that each person will require 5Mbytes...

Would it be ok for me to use the PERT formula (O + 4M + P)/6 and conclude ((.25 + 4(1) +5)/6 = 1.54) that my storage needs will be 40,000x1.5 = 60,000 Mbytes = 60 Gigabytes ?

I am trying to reach the answers "How likely would that be? ¿How can I know if that is X% likely?" numbers using Mathematica, so far it seems I can answer: How likely would that be? by writing this:

Probability[x < 1.54, x [Distributed] PERTDistribution[{0.25, 5}, 1]]

I get a 0.55 probability of X being smaller than 1.54. ¿Correct?

I am asking this because I need to estimate the hardware requirements for a software system, and all the information I am given is that "I am going to store the information of 10,000 people each year for 4 years, that is 40,000 files". Once the system works for some months I will be able to use its behavior to predict its needs in a more accurately way, but in the meanwhile, I need to be able to give an initial estimate... and I am not sure on how to do that...

So. would it be safe to say that I have a 0.55 probability of my storage needs being smaller than 60 Gigabytes?

How to estimate storage needs? I am using the PERT Distribution for filesizes ¿How to aggregate it?

Lets say I know that I am going to store the information of 10,000 people each year for 4 years, that is 40,000 files. Now If I estimate that on the best case scenario the information from each person is going to weight .25MBytes, I believe there is a 50% chance that it will weight 1Mbyte, and I believe that the worst case scenario is that each person will require 5Mbytes...

Would it be ok for me to use the PERT formula (O + 4M + P)/6 and conclude ((.25 + 4(1) +5)/6 = 1.54) that my storage needs will be 40,000x1.5 = 60,000 Mbytes = 60 Gigabytes ?

I am trying to reach the answers "How likely would that be? ¿How can I know if that is X% likely?" numbers using Mathematica, so far it seems I can answer: How likely would that be? by writing this:

$Probability[x < 1.54, x \[Distributed] PERTDistribution[{0.25, 5}, 1]]$

I get a 0.55 probability of X being smaller than 1.54. ¿Correct?

I am asking this because I need to estimate the hardware requirements for a software system, and all the information I am given is that "I am going to store the information of 10,000 people each year for 4 years, that is 40,000 files". Once the system works for some months I will be able to use its behavior to predict its needs in a more accurately way, but in the meanwhile, I need to be able to give an initial estimate... and I am not sure on how to do that...

So. would it be safe to say that I have a 0.55 probability of my storage needs being smaller than 60 Gigabytes?

I am having some trouble beliving this is right, so I made the follwing experiment:

$Table[Fold[Plus, 0, RandomVariate[PERTDistribution[{.25, 5}, 1], 40000]], {10000}] $

To simulate the total size that I would need and, the thing is that this giving me a different answer than I would think:

If I do this:

$Quantile[Table[Fold[Plus, 0, RandomVariate[PERTDistribution[{.25, 5}, 1], 40000]], {10000}], 95/100]$

Answer: $61930.7$

It should give me the same answer as this (Should'nt it?) why it doesn't?

$(Quantile[PERTDistribution[{.25, 5}, 1], 95/100]) (40000)$

Answer: $121,938$

But it does not, it seems like estimating a single files size with 95% confidence and then multiplying it for 40,000 is very different... ¿where is the mistake? also ¿is there a way to deal with this mistake "symbolically" that is, without having to wait for slow generation of sizes?

deleted 30 characters in body
Source Link
Luxspes
  • 301
  • 2
  • 10
Loading
added 2 characters in body
Source Link
Luxspes
  • 301
  • 2
  • 10
Loading
edited body; edited tags; edited title
Source Link
Luxspes
  • 301
  • 2
  • 10
Loading
More detail, some calculations added, making more effort to answer my own question, but still not sure of the answer
Source Link
Luxspes
  • 301
  • 2
  • 10
Loading
added 474 characters in body
Source Link
Luxspes
  • 301
  • 2
  • 10
Loading
Source Link
Luxspes
  • 301
  • 2
  • 10
Loading